Auto-analyze when sequencing has ended

[Nicklas Nordborg]

Reggie should be able to automatically detect when a sequencing run has ended and, after making some checks, start the demux step and other downstream secondary analysis.

The "Register sequencing started" wizard should have an option to enable the automatic analysis. It should be something that is similar to the "auto-confirm" option in the "Start demux" wizard.

Implementation details:

• When the auto-analyze option is enabled the SequencingRun item is connected with a Job (external).
• A signal handler is used for status updates. We need to implement a new handler that is similar to the Open Grid Scheduler handler.
• The signal handler will check the <run-archive> for certain files and update the job status based on which files that are present:
  • no files found --> WAITING
  • Data folder found --> EXECUTING (clustering)
  • RunParameters.xml --> EXECUTING (sequencing)
  • RunCompletionStatus.xml --> DONE
  • (more investigations may be needed)
• When the sequencing has completed it should import data from the files just as if the "Register sequencing ended" wizard had been used.
• It should always be possible to run the "Register sequencing ended" wizard manually.
• The auto-confirmation service will then take over and detect the ended job. It should run either picard or the genseq_check_illumina_dir.pl program for a preliminary check.
• If all seems to be ok, the demux step should be started using the default options (including auto-confirm=true).

[Nicklas Nordborg]

(In [3691]) References #834: Auto-analyze when sequencing has ended

Added "auto-analyze" option to the "Register sequencing started" wizard. When enabled, a cluster from the configuration file should be selected. It is also possible to select job priority and debug mode (eg. the same parameters as in the "Start demux" wizard).

The parameters are currently ignored by the registration.

[Nicklas Nordborg]

(In [3692]) References #834: Auto-analyze when sequencing has ended

Sequencing runs with "auto-analyze" enabled are now saved with a job item linked to them. Signal processing is not implemented so the job will just sit in "waiting" status.

The "Register sequencing ended" wizard now display a warning if trying to maually register a sequencing run with "auto-analyze" as ended.

[Nicklas Nordborg]

(In [3697]) References #834: Auto-analyze when sequencing has ended

Implemented progress reporting for sequencing jobs. The signal handler implementation now accepts a flow cell barcode. The status checking has been refactored to make it possible to call different scripts for different type of jobs. The NextSeqJobStatusReuqest uses a new pipeline script nextseq_status.sh that check which files and folders for a given flow cell that exists in the run archive.

[Nicklas Nordborg]

(In [3699]) References #834: Auto-analyze when sequencing has ended

Implemented job completion handler for sequencing runs. The completion handler will download the RunParameters.xml and extract the same information as the "Register sequencing ended" wizard. After the completion handler has ended the sequencing run should end up in the "Confirm sequencing completed" wizard.

[Nicklas Nordborg]

(In [3700]) References #834: Auto-analyze when sequencing has ended

Started to implement SequencingRunAutoConfirmer. It will currently accept and confirm sequencing runs that has ended with Job.Status=DONE. Checking with Picard is not yet implemented.

Starting the demux step is also not implmented so the sequencing run will end up in "Start demux" wizard.

[Nicklas Nordborg]

(In [3704]) References #834: Auto-analyze when sequencing has ended

Implemented check with Picard.

[Nicklas Nordborg]

(In [3714]) References #834: Auto-analyze when sequencing has ended

Re-factored the code for submitting demux jobs to the cluster. More or less everything is in place, except that the barcode files needed by Picard are not genered. This is due to the auto-confirmation needs to work in a single transaction, but the current barcode files exporter require the data to committed first.

[Nicklas Nordborg]

(In [3715]) References #834: Auto-analyze when sequencing has ended

Auto-confirmation is now starting the demux job.

[Nicklas Nordborg]

(In [3716]) References #834: Auto-analyze when sequencing has ended

The barcode files exporter for the picard demux step has been fixed so that it works with uncommitted items. The demux step should now be working again and the auto-confirm functionality is fully implemented. It only remains to clean up the code and make some final tests.

[Nicklas Nordborg]

(In [3717]) References #834: Auto-analyze when sequencing has ended

Removing debug output (found some left-overs in NoSpecimen as well).

The 'debug' option is selected by default on developer installations.

[Nicklas Nordborg]

(In [3718]) References #834: Auto-analyze when sequencing has ended

The "Register sequencing ended" wizard now display some more information about sequencing runs that have auto-analyze enabled. It is still possible to manually register the sequencing as ended (and disabled auto-analyze) but a warning is displayed.