Changes between Version 3 and Version 4 of net.sf.basedb.opengrid/using

Timestamp:

Jan 13, 2017, 3:06:17 PM (7 years ago)

Author:

Nicklas Nordborg

Comment:

Started to write the section about "Getting notified when a job completes"

net.sf.basedb.opengrid/using

@@ -115 +115 @@
 Job job = Job.getNew(dc, null, null, null); // All null to create an 'OTHER' type job
 job.setName("My analysis");
-job.setPluginVersion("analysis-1.0");
+job.setPluginVersion("my-analysis-1.0");
 // job.setItemSubtype(...); // This can also be useful
 dc.saveItem(job); // Important!!!
@@ -149 +149 @@
 == Getting notified when a job completes ==
 
+One important feature is that other extensions can get notified when a job running on the cluster has ended. This is implemented in an asynchronous manner and it should not matter if the BASE server is updated or restarted or otherwise modified while a job is running. In the background there are two parts that work together to make this feature work.
+
+ * The BASE system for requesting job progress information about external jobs has been setup to send requests to the `OpenGridService` whenever it want new information about a job. This is the reason why it is important to create a BASE job item as a proxy for the Open Grid Cluster jobs. Without it no progress information is requested and we never get to know when the job has ended.
+ * The `OpenGridService` is polling each registered cluster at regular intervals. Typically once every minute but it may be more or less often depending on if there are any known jobs executing or not. The `OpenGridSession.qstat()` and `OpenGridSession.qacct()` methods are used for this and will detect waiting, running and completed jobs. For running jobs, the service will download the `progress` file (see `ScriptBuilder.progress()` above) and about the information in the BASE database.
+
+Once a job has been detected as completed the service will invoke the job completion sequence. This is implemented as a custom extension point (`net.sf.basedb.opengrid.job-complete`) that will receive messages about completed jobs. Extensions that want to get notified should extend the extension point. Note that all registered extensions are notified about all jobs. It doesn't matter which extension that originally submitted the job to the cluster. Notifications are sent both for successful and failed jobs. Thus, each extension is responsible for filtering and ignoring notifications about jobs that is of no interest to them. This is why it is important to set name, plug-in version, etc. on the job when submitting it. We recommend that this filtering step is implemented in the `ActionFactory` that is registered for the `net.sf.basedb.opengrid.job-complete` extension point. Note that a single notification may handle more than one job. Thus, the `prepareContext()` method is called once and without any information about the jobs while the the `getActions()` method is called once for every job.
+
+{{{
+public class MyAnalysisJobCompletionHandlerFactory 
+   implements ActionFactory<JobCompletionHandler>
+{
+        
+   public MyAnalysisJobCompletionHandlerFactory()
+   {}
+
+   @Override
+   public boolean prepareContext(InvokationContext context) 
+   {
+      // Always true since we do not know anything about the job(s) that have been completed
+      return true;
+   }
+
+   @Override
+   public JobCompletionHandler[] getActions(InvokationContext context) 
+   {
+      ClientContext cc = context.getClientContext();
+      Job job  = (Job)cc.getCurrentItem();
+                
+      String pluginVersion = job.getPluginVersion();
+      if (pluginVersion == null || !pluginVersion.startsWith("my-analysis"))
+      {
+         // This is not our job, ignore it
+         return null;
+      }
+                
+      // Note that job.getStatus() has not been updated yet so we
+      // need to get the status information extracted from the cluster
+      JobStatus status = (JobStatus)cc.getAttribute("job-status");
+      if (status.getStatus() != Job.Status.DONE)
+      {
+         // We don't do anything unless the job was successful.
+         return null;
+      }
+
+      JobCompletionHandler action = null;
+      String jobName = job.getName();
+      if (jobName.startsWith("My analysis"))
+      {
+         action = new MyAnalysisCompletionHandler();
+      }
+      else
+      {
+          // In the future we may have more than one type of jobs...
+      }
+
+      return action == null ? null : new JobCompletionHandler[] { new JobCompletionWrapper(action) };
+   }
+}
+}}}
+
 == Aborting jobs ==