Move Open Grid Scheduler functionality to a separate extension

[Nicklas Nordborg]

See #904.

[Nicklas Nordborg]

(In [4259]) References #905: Move Open Grid Scheduler functionality to a separate extension

Started to test this. Existing functionality that depends on current SSH implementation in Reggie has been disabled. This includes reading reggie-config.xml. The "Access remote hosts including Open Grid Scheduler hosts" wizard has been update to display and use information from the OpenGrid extension. I think this functionality should be moved out of Reggie and instead be included in the OpenGrid extension, but that will have to wait. Next step is to try to actually submit a job to the cluster. A big issue here is what to do with the configuration in reggie-config.xml. It needs a slightly different approach now that cluster information is in opengrid-config.xml but we do not want to store Reggie-specific information in there.

[Nicklas Nordborg]

(In [4266]) References #905: Move Open Grid Scheduler functionality to a separate extension

Started to re-design the configuration file. Basically each <host> entry now only need to reference an entry in the opengrid-config.xml file. Login information and paths to job folders are no longer needed.

The "Remote hosts" listing now include a column if a Reggie configuration has been defined for that host or not. The "Reload configuration" is semi-functional again. It can reload the reggie-config.xml file but not the opengrid-config.xml file.

This change should be enough to make it possible to move to use the Open Grid extensions for submitting jobs, but I think the configuration needs some more flexibility (for example, using the same configuration for mor than one cluster).

[Nicklas Nordborg]

(In [4268]) References #905: Move Open Grid Scheduler functionality to a separate extension

Updated the AlignJobCreator to use the Open Grid extension for submitting jobs. It seems to work but reading back the results and auto-confirmation have not yet been implemented. If auto-confirm has been checked it will currently leave the item in a limbo state so that one has to dig around a bit in order to be able to do anything more with it.

[Nicklas Nordborg]

(In [4271]) References #905: Move Open Grid Scheduler functionality to a separate extension

Job completion is now supported for alignment jobs.

[Nicklas Nordborg]

(In [4272]) References #905: Move Open Grid Scheduler functionality to a separate extension

Updated the CufflinksJobCreator to use the Open Grid extension for submitting jobs. Job completion handling (which in this case means creating a job for importing the isoforms.tracking_fpkm file) also seems to work. Auto-confirmation has not been implemented yet.

[Nicklas Nordborg]

(In [4273]) References #905: Move Open Grid Scheduler functionality to a separate extension

Enabled auto-confirmation for demux and align jobs. Not tested yet.

[Nicklas Nordborg]

(In [4274]) References #905: Move Open Grid Scheduler functionality to a separate extension

Updated the DemuxJobCreator to use the Open Grid extension for submitting jobs. The 'number of slots' input field is not populated since there is not yet any way to send the configuration. The 'Check data files' function is not implemented. The auto-confirm should work, but it has not been tested.

[Nicklas Nordborg]

(In [4276]) References #905: Move Open Grid Scheduler functionality to a separate extension

The "Check data files" should now be working.

[Nicklas Nordborg]

(In [4277]) References #905: Move Open Grid Scheduler functionality to a separate extension

Searching the run archive for RunParameters.xml file from the Register sequencing ended wizard should now work.

[Nicklas Nordborg]

(In [4279]) References #905: Move Open Grid Scheduler functionality to a separate extension

Updated the "Stray files wizard" to use the functionality in the Open Grid package now. We needed the FileServer support for this.

The function for viewing files in a remote folder has also been updated. It should now work with both results files from a job and files in the project archive.

[Nicklas Nordborg]

(In [4280]) References #905: Move Open Grid Scheduler functionality to a separate extension

Starting to remove files that are no longer needed and marked a lot of other files as deprecated. This makes it easier to find remaing functionality that depend on Open Grid/SSH access to remote servers.

[Nicklas Nordborg]

(In [4281]) References #905: Move Open Grid Scheduler functionality to a separate extension

The ProjectArchiveService that is responsible for setting access permissions on files in the project archive based on the consent information found on the associated case has been updated.

Also added ScriptUtil and moved the setUmaskForItem() helper methdo there. So now the file permissions should be set and updated correctly.

[Nicklas Nordborg]

(In [4282]) References #905: Move Open Grid Scheduler functionality to a separate extension

The RetractionServlet has been updated to use the Open Grid package for removing files from the project archive.

[Nicklas Nordborg]

(In [4283]) References #905: Move Open Grid Scheduler functionality to a separate extension

It is now possible to use the "Sequencing started" wizard again. The "Auto-analyze" function doesn't work. The "hack" with pick-backing on the Open Grid job status updater that we used before to check the progress of the sequencer can't be used any longer. To fix this we need to implement our own signal handler extension.

[Nicklas Nordborg]

(In [4285]) References #905: Move Open Grid Scheduler functionality to a separate extension

Added the ReggieSignalHandlerFactory and NextSeqSignalHandler. The factory will respond to signals with the reggie: scheme and dsipatch to a proper signal handler. Currently the NextSeq signal handler is the only one that is used. When a status update request is coming in the handler will register itself as an aync callback with the OpenGridService.

When the OpenGridService is doing the next check it will call the NextSeqSignalHandler that checks the status of the sequencing run and reports that back to Open Grid service. The service will update the job information if needed. Once the sequenincing has been completed the regular job completion handling should be invoked but this has not yet been tested.

[Nicklas Nordborg]

(In [4286]) References #905: Move Open Grid Scheduler functionality to a separate extension

Removed the remaining old Open Grid code. Most features are now moved to the new API provided by the Open Grid extension. There are probably a few thing to polish up and more testing is needed to make sure that the entire chain is working (particularly the auto-analyze/auto-confirm code).

[Nicklas Nordborg]

(In [4287]) References #905: Move Open Grid Scheduler functionality to a separate extension

Consolidating code by moving some of the remaining functionality (mostly utility functions) to fewer classes and packages.

[Nicklas Nordborg]

(In [4288]) References #905: Move Open Grid Scheduler functionality to a separate extension

Moving job creator code to a different package.

[Nicklas Nordborg]

(In [4289]) References #905: Move Open Grid Scheduler functionality to a separate extension

Loading pre-configured job priorities and extra configuration settings when they are requested (the demux wizard need the number of slots setting).

[Nicklas Nordborg]

(In [4291]) References #905: Move Open Grid Scheduler functionality to a separate extension

Implementing a wrapper around job completion handlers to make sure that the counter and auto-confirm services are updated as soon as possible in a consistent manner.

Removed some debug output.

[Nicklas Nordborg]

(In [4303]) References #905: Move Open Grid Scheduler functionality to a separate extension

Next release requries BASE 3.10. Checked in the opengrid.jar file. It should be replaced once the Open Grid extension has been relased.

[Nicklas Nordborg]

(In [4306]) References #905: Move Open Grid Scheduler functionality to a separate extension

Merge /branches/ticket-905 to the trunk.

[Nicklas Nordborg]

(In [4308]) References #905: Move Open Grid Scheduler functionality to a separate extension

Renamed the "Open Grid" wizard.

Clarified the documentation about the ID that is needed to reference the Open Grid cluster.

[Nicklas Nordborg]

(In [4311]) References #905: Move Open Grid Scheduler functionality to a separate extension

Fixed problems with error handling when submitting jobs failed on the Open Grid cluster side. This typically resulted in a NullPointerException later in the code. The problem should now be detected earlier and throw an exception with a better error message.

[Nicklas Nordborg]

(In [4313]) References #905: Move Open Grid Scheduler functionality to a separate extension

Register an event handler that detects when the Open Grid service is started and then forces a reload of the reggie configuration file.

[Nicklas Nordborg]

(In [4316]) References #905: Move Open Grid Scheduler functionality to a separate extension

Removed the possibility to manually execute commands via SSH on the Open Grid clusters.

The list with Open Grid Clusters in Reggie is linked to the corresponding overview page that is included in the Open Grid extension.

[Nicklas Nordborg]

(In [4322]) References #905: Move Open Grid Scheduler functionality to a separate extension

Updated to Open Grid 1.0-rc1.