Changes between Version 8 and Version 9 of net.sf.basedb.opengrid/using


Ignore:
Timestamp:
Jan 16, 2017, 3:04:04 PM (3 years ago)
Author:
Nicklas Nordborg
Comment:

Added example with non-Open Grid jobs

Legend:

Unmodified
Added
Removed
Modified
  • net.sf.basedb.opengrid/using

    v8 v9  
    260260
    261261Tip! It is possible to create `ConnectionInfo` instance from a BASE `FileServer` item (assuming that the file server contains all required information for connecting via SSH: host, fingerprint, username and password).
     262
     263=== Tracking non-Open Grid jobs ===
     264
     265Sometimes there are other things going on that are not Open Grid jobs that would be interesting to track. One example is the sequencing progress of a sequencer machine. In this case we want to know when the sequencing has been completed and then start analysis jobs (as Open Grid Cluster jobs). A simple bash script has been implemented (http://baseplugins.thep.lu.se/browser/other/pipeline/trunk/nextseq_status.sh) that checks if all result files from the sequencing are present on the file server or not. We want to run this script at regular intervals and then when all data is present, run some checks and start the analysis jobs. There are three steps to consider altogether:
     266
     267 * The sequencing process should be represented by a BASE job item as a proxy. Progress reporting need to be setup using the extension mechanism implemented in the BASE core. This need to implemented completely by the other extension. It is not possible to re-use the setup that this extension is using.
     268
     269{{{
     270String barcode = ... // Something that identifies the current sequencing
     271String clusterId = ... // The Open Grid Cluster we will use to check status
     272
     273// Create a new BASE job and set properties so that we can identify it later
     274Job job = Job.getNew(dc, null, null, null); // All null to create an 'OTHER' type job
     275job.setName("My sequencing");
     276job.setPluginVersion("my-sequencing-1.0");
     277
     278job.setExternalId(barcode);  // Instead of the Open Grid job ID
     279// Setup signalling for progress reporting (see BASE documentation)
     280String signalURI = SequencingSignalHandler.getSignalUri(barcode);
     281job.setSignalTransporter(ExtensionSignalTransporter.class, signalURI);
     282
     283// We need to know which cluster to use
     284job.setScheduled(clusterId, null);
     285dc.saveItem(job); // Important!!!
     286}}}
     287
     288 * Once a request for a status update is received by the handler it should call `OpenGridService.asyncJobStatusUpdate(JobIdentifier, JobStatusUpdater)`. The Open Grid extension will then call the `JobStatusUpdater.getJobStatus()` during the next async processing cycle. In the example above, the `JobStatusUpdater` implementation should call the bash script to see how far the sequencing has come and then report that back in a `JobStatus` object. The Open Grid extension will take the responsibility of updating the Job item in BASE.
     289
     290{{{
     291public class SequencingSignalHandler
     292   implements SignalHandler, JobStatusUpdater
     293{
     294
     295   // Not all SignalHandler methods are shown...
     296
     297   @Override
     298   public void handleSignal(Signal signal)
     299   {
     300      String barcode = job.getExternalId();
     301      String clusterId = job.getServer();
     302               
     303      JobIdentifier jobId = new JobIdentifier(clusterId, flowCellId, job.getId());
     304      OpenGridService.getInstance().asyncJobStatusUpdate(jobId, this);
     305   }
     306
     307   // Execute the bash script and parse out the result to a JobStatus object
     308   // The code shown here has been simplified...
     309   @Override
     310   public JobStatus getJobStatus(OpenGridSession session, JobIdentifier jobId)
     311   {
     312      String barcode = jobId.getClusterJobId();
     313               
     314      CmdResult<String> sequencingStatus = session.executeCmd("nextseq_status.sh " + barcode, 5);
     315      if (sequencingStatus .getExitStatus() != 0)
     316      {
     317         logger.error("nextseq_status.sh failed: " + sequencingStatus.toString());
     318         return null;
     319      }
     320
     321      JobStatus status = null;
     322      try
     323      {
     324         status = parseSequencingStatus(sequencingStatus.getStdout());
     325      }
     326      catch (Exception ex)
     327      {
     328         logger.error("Could not parse nextseq_status.sh output", ex);
     329      }
     330      return status;
     331   }
     332}
     333}}}
     334
     335 * When the sequencing has been completed (`JobStatus == DONE`) this is detected using the normal job completion routines in the Open Grid extension which will notify all registered `JobCompletionHandler` implementations. The other extension simply need to extend the `JobCompletionHandler` implementation to be able to detect the sequencing job and then do whatever needs to be done with that.
     336
     337{{{
     338public class MySequencingJobCompletionHandlerFactory
     339   implements ActionFactory<JobCompletionHandler>
     340{
     341       
     342   ....
     343
     344   @Override
     345   public JobCompletionHandler[] getActions(InvokationContext context)
     346   {
     347      ....
     348               
     349      String pluginVersion = job.getPluginVersion();
     350      if (pluginVersion != null && !pluginVersion.startsWith("my-sequencing"))
     351      {
     352         // This is our sequencing job
     353         action = new SequencingJobCompletionHandler();
     354      }
     355               
     356      ....
     357   }
     358}
     359}}}
     360
     361