Changes between Version 8 and Version 9 of net.sf.basedb.opengrid/using

Timestamp:

Jan 16, 2017, 3:04:04 PM (8 years ago)

Author:

Nicklas Nordborg

Comment:

Added example with non-Open Grid jobs

net.sf.basedb.opengrid/using

@@ -260 +260 @@
 
 Tip! It is possible to create `ConnectionInfo` instance from a BASE `FileServer` item (assuming that the file server contains all required information for connecting via SSH: host, fingerprint, username and password).
+
+=== Tracking non-Open Grid jobs ===
+
+Sometimes there are other things going on that are not Open Grid jobs that would be interesting to track. One example is the sequencing progress of a sequencer machine. In this case we want to know when the sequencing has been completed and then start analysis jobs (as Open Grid Cluster jobs). A simple bash script has been implemented (http://baseplugins.thep.lu.se/browser/other/pipeline/trunk/nextseq_status.sh) that checks if all result files from the sequencing are present on the file server or not. We want to run this script at regular intervals and then when all data is present, run some checks and start the analysis jobs. There are three steps to consider altogether:
+
+ * The sequencing process should be represented by a BASE job item as a proxy. Progress reporting need to be setup using the extension mechanism implemented in the BASE core. This need to implemented completely by the other extension. It is not possible to re-use the setup that this extension is using.
+
+{{{
+String barcode = ... // Something that identifies the current sequencing 
+String clusterId = ... // The Open Grid Cluster we will use to check status 
+
+// Create a new BASE job and set properties so that we can identify it later
+Job job = Job.getNew(dc, null, null, null); // All null to create an 'OTHER' type job
+job.setName("My sequencing");
+job.setPluginVersion("my-sequencing-1.0");
+
+job.setExternalId(barcode);  // Instead of the Open Grid job ID
+// Setup signalling for progress reporting (see BASE documentation)
+String signalURI = SequencingSignalHandler.getSignalUri(barcode);
+job.setSignalTransporter(ExtensionSignalTransporter.class, signalURI);
+
+// We need to know which cluster to use
+job.setScheduled(clusterId, null);
+dc.saveItem(job); // Important!!!
+}}}
+
+ * Once a request for a status update is received by the handler it should call `OpenGridService.asyncJobStatusUpdate(JobIdentifier, JobStatusUpdater)`. The Open Grid extension will then call the `JobStatusUpdater.getJobStatus()` during the next async processing cycle. In the example above, the `JobStatusUpdater` implementation should call the bash script to see how far the sequencing has come and then report that back in a `JobStatus` object. The Open Grid extension will take the responsibility of updating the Job item in BASE.
+
+{{{
+public class SequencingSignalHandler
+   implements SignalHandler, JobStatusUpdater
+{
+
+   // Not all SignalHandler methods are shown...
+
+   @Override
+   public void handleSignal(Signal signal) 
+   {
+      String barcode = job.getExternalId();
+      String clusterId = job.getServer();
+                
+      JobIdentifier jobId = new JobIdentifier(clusterId, flowCellId, job.getId());
+      OpenGridService.getInstance().asyncJobStatusUpdate(jobId, this);
+   }
+
+   // Execute the bash script and parse out the result to a JobStatus object
+   // The code shown here has been simplified...
+   @Override
+   public JobStatus getJobStatus(OpenGridSession session, JobIdentifier jobId)
+   {
+      String barcode = jobId.getClusterJobId();
+                
+      CmdResult<String> sequencingStatus = session.executeCmd("nextseq_status.sh " + barcode, 5);
+      if (sequencingStatus .getExitStatus() != 0)
+      {
+         logger.error("nextseq_status.sh failed: " + sequencingStatus.toString());
+         return null;
+      }
+
+      JobStatus status = null;
+      try
+      {
+         status = parseSequencingStatus(sequencingStatus.getStdout());
+      }
+      catch (Exception ex)
+      {
+         logger.error("Could not parse nextseq_status.sh output", ex);
+      }
+      return status;
+   }
+}
+}}}
+
+ * When the sequencing has been completed (`JobStatus == DONE`) this is detected using the normal job completion routines in the Open Grid extension which will notify all registered `JobCompletionHandler` implementations. The other extension simply need to extend the `JobCompletionHandler` implementation to be able to detect the sequencing job and then do whatever needs to be done with that.
+
+{{{
+public class MySequencingJobCompletionHandlerFactory 
+   implements ActionFactory<JobCompletionHandler>
+{
+        
+   ....
+
+   @Override
+   public JobCompletionHandler[] getActions(InvokationContext context) 
+   {
+      ....
+                
+      String pluginVersion = job.getPluginVersion();
+      if (pluginVersion != null && !pluginVersion.startsWith("my-sequencing"))
+      {
+         // This is our sequencing job
+         action = new SequencingJobCompletionHandler();
+      }
+                
+      ....
+   }
+}
+}}}
+
+