Implement support for running jobs in a Singularity container

[Nicklas Nordborg]

In a Singularity container we can install all programs that are needed for a specific analysis step without having to worry about exactly which versions that are installed on the host system. In the simplest form a script can be started with:

singularity exec /path/to/container.sif job.sh

but we will at least also need to specify --bind options so that the container can access project-archive, run-archive, and other directories where we store data.

We also need to split the job.sh script, since we need the options for qsub. For example:

start_job.sh is submitted to the queue system and contains the options is responsible for starting the job.sh script:

# --- start of qsub options ---
#$ -S /bin/sh
#$ -terse
....
# --- end of qsub options ---
singularity exec /path/to/container.sif job.sh

Also, since the container typically know and have control over the paths were the programs are installed we should not have to specify them in the configuration file.

Note that we still want to support the non-container way of running jobs.

Here is the final solution

The submission procedure has changed quite a lot and there are now several scripts. Scripts that are dynamically generated are designed to be as small as possible and should in most cases only define environment variables and then call a static script that performs the actual work.

• submit.sh: This is a generated script and contain options for the job scheduler. It will start the run.sh script.
• run.sh: This is a static script that is mostly about error handling and is slightly different depending on the cluster type. It will start the job.sh script.
• job.sh: This is a generated script with all options for running the selected analysis. It will setup paths to data folders, program parameters, etc. and then run another static script that is doing the actual analysis.

The job.sh script is generated by a combination of settings in reggie-config.xml and information that is stored in the database. There are major changes in the reggie-config.xml, but all analysis steps are now very similar in how they are configured:

• <global-env>: Intended for setting up paths to common folders. For example the <reference-folder> is now defined here as export ReferenceFolder=.... This setting is included in job.sh for all analysis steps.

• <..>/<env>: section that is intended for setting up environment variables for a specific analsysis. Lots of settings that were configured by subtags show now go in this sections instead.
• <..>/<env-debug>: this section is included if the job is running in debug mode.
• <..>/<execute>: the command for executing the analysis script. This is typically as simple as ./run-analysis.sh but can be more complex if the analsyis should run in a container.

Here is an example for the StringTie step:

<stringtie>
  <env>
    export STRINGTIE="/usr/local/packages/stringtie/1.3.3b/bin/stringtie"
    export PREPDE="/usr/local/packages/stringtie/1.3.3b/bin/prepDE.py"
    export GTF="${ReferenceFolder}/hg38/hg38.analysisSet_gencode27_snp150/gencode.v27.primary_assembly.annotation_subset_transcripttype_proteincoding.gtf"
    export StringTieOptions="--rf -B -e"
  </env>
  <env-debug></env-debug>
  <execute>./stringtie.sh</execute>
</stringtie>

When running in a container we do not need to specify paths to programs (since the container already knows this):

<stringtie>
  <env>
    export GTF="${ReferenceFolder}/hg38/hg38.analysisSet_gencode27_snp150/gencode.v27.primary_assembly.annotation_subset_transcripttype_proteincoding.gtf"
    export StringTieOptions="--rf -B -e"
  </env>
  <env-debug></env-debug>
  <execute>
    singularity exec --bind ${ArchiveFolder},${ReferenceFolder} ${ContainerFolder}/stringtie-v1-stringtie1.3.3b.sif ./stringtie.sh
  </execute>
</stringtie>

[Nicklas Nordborg]

In 6626:

References #1374: Implement support for running jobs in a Singularity container

Adding more environment variables to the mbaf_setup.sh script. The intention is to make the mbaf.sh script as static as possible and to lift that out as a separate script file that doesn't have to be created by java code.

[Nicklas Nordborg]

In 6628:

References #1374: Implement support for running jobs in a Singularity container

The mBaf analysis step should now support running inside a container. To do that the mbaf/execute option must be configured in reggie-config.xml. Example:

<execute>
singularity exec --bind ${ArchiveRoot},${ReferenceFolder} ${ContainerFolder}/mbaf.sif ./mbaf.sh
</execute>

The mbaf.sif container need Java (1.8) and GATK (3.8).

[Nicklas Nordborg]

In 6630:

References #1374: Implement support for running jobs in a Singularity container

Change scripts to use '/bin/bash' instead of '/bin/sh'.

[Nicklas Nordborg]

In 6631:

References #1374: Implement support for running jobs in a Singularity container

The StringTie pipeline has now been updated to work with containers. The stringtie/execute option must be set to a command for starting the container. Example: singularity exec --bind ${ArchiveRoot},${ReferenceFolder} ${ContainerFolder}/stringtie.sif ./stringtie.sh

[Nicklas Nordborg]

In 6638:

References #1374: Implement support for running jobs in a Singularity container

Updated the MIPS-related pipelines to not use deprecated API from the opengrid package. Since MIPS is not used today the actual pipelines will not be modified for use with singularity.

[Nicklas Nordborg]

In 6640:

References #1374: Implement support for running jobs in a Singularity container

The Hisat alignment step has been updated.

[Nicklas Nordborg]

In 6649:

References #1374: Implement support for running jobs in a Singularity container

The RNAseq demux step is now updated.

[Nicklas Nordborg]

In 6653:

References #1374: Implement support for running jobs in a Singularity container

Re-factored scripts and configuration settings which should make things more easy to setup.

In the configuration file a lot of special tags have been removed and replaced with a <env> setting. This setting is added to the job script as is and is intended to be used for setting up environment variables that are needed by the script.
Note that this section should also define paths to programs. The <programs> sections is going to be removed.

When running in a singularity container the paths are already defined by the container and they need not be defined in the <env> section. The container will typically ignore them anyway unless they are specified with the --env option or SINGULARITYENV_ prefix.

For example: singularity ... --env "JAVA=$JAVA" or SINGULARITYENV_JAVA=$JAVA must be used to override the JAVA variable defined inside the container.

[Nicklas Nordborg]

In 6654:

References #1374: Implement support for running jobs in a Singularity container

Re-factored the Hisat align step.

[Nicklas Nordborg]

In 6655:

References #1374: Implement support for running jobs in a Singularity container

Re-factored the RNAseq demux step.

[Nicklas Nordborg]

In 6656:

References #1374: Implement support for running jobs in a Singularity container

The Variant calling pipeline is now updated.

[Nicklas Nordborg]

In 6657:

References #1374: Implement support for running jobs in a Singularity container

The Targeted genotyping pipeline has been updated.

[Nicklas Nordborg]

In 6658:

References #1374: Implement support for running jobs in a Singularity container

The Import FASTQ pipeline has been updated.

[Nicklas Nordborg]

In 6661:

References #1374: Implement support for running jobs in a Singularity container

Implemented support for running the "Data check".

[Nicklas Nordborg]

In 6662:

References #1374: Implement support for running jobs in a Singularity container

Scripts for checking sequencing status has been moved into Reggie. Not really needed for Singularity containers since they are not using that functionality. Main purpose is to get rid of the external pipeline scripts.

[Nicklas Nordborg]

In 6663:

References #1374: Implement support for running jobs in a Singularity container

The code for getting information from runParameters.xml has been implemented as a script instead of auto-generated by the java code.

[Nicklas Nordborg]

In 6665:

References #1374: Implement support for running jobs in a Singularity container

Some more things that has been cleaned up:

• <env-debug> setting for all pipelines.
• Removed use of deprecated methods in JobConfig (replaced with call to set -e).
• Added environment variable $JavaOptions to scripts that use java so that it is possible to configure command line parameters for java if needed.
• Configuration file cleaned up and prepared with settings suitable for the current setup.
• Some other minor fixes.

[Nicklas Nordborg]

In 6669:

References #1374: Implement support for running jobs in a Singularity container

Changed scripts to make them easier to use for debugging. Each step in a script is now checking if a specfic file exists in the done/ folder before starting that part of the script. If the file doesn't exists, the step is executed. The last part of the step create the .done file. If the file already exists the step is skipped.

For example, efter copying BAM files to the working directory, done/copy.done is created. If the scripts fails at the next step it is possible to modify and re-try the script (manually) and the copy step will be skipped since the done/copy.done file already exists. Since some steps are time-consuming this will save a lot of time when developing and debugging.

[Nicklas Nordborg]

In 6673:

References #1374: Implement support for running jobs in a Singularity container

StringTie jobs used the incorrect environment options when debugging.

[Nicklas Nordborg]

In 6675:

References #1374: Implement support for running jobs in a Singularity container

Updated the legacy pipeline.

[Nicklas Nordborg]

In 6676:

References #1374: Implement support for running jobs in a Singularity container

Found some minor issues in configuration.

[Nicklas Nordborg]

In 6677:

References #1374: Implement support for running jobs in a Singularity container

Added several Singularity container definition files in the /containers folder.

• hisat.def: Container with the software that is needed for the Hisat alignment step
• mbaf.def: Container with the software that is needed for the mBAF analysis step
• stringtie.def: Container with the software that is needed for the Stringtie gene expression analysis step

[Nicklas Nordborg]

In 6679:

References #1374: Implement support for running jobs in a Singularity container

Added more Singularity container definition files in the /containers folder.

• rnaseq-demux.def: Container with the software that is needed for the Demux and FASTQ import steps
• variantcall.def: Container with the software that is needed for the Variant calling step
• rockylinux8.4_with_utils.def: A container with Rocky Linux and some basic utilities. Useful for getting started with a new build.
• rockylinux8.4_with_miniconda.def: A container with Rocky Linux and Miniconda. Useful for gettings started with a new build. There are alternate versions of Miniconda to select from. Edit the file as needed. Check ​https://repo.anaconda.com/miniconda/ for even more versions!

[Nicklas Nordborg]

In 6680:

References #1374: Implement support for running jobs in a Singularity container

Added more Singularity container definition file for the targeted genotyping analysis.

[Nicklas Nordborg]

In 6681:

References #1374: Implement support for running jobs in a Singularity container

Added a Singularity container definition file for the legacy pipeline.

[Nicklas Nordborg]

In 6682:

References #1374: Implement support for running jobs in a Singularity container

Changed "Open Grid scheduler" to "Job scheduler" in several places.

[Nicklas Nordborg]

In 6685:

References #1374: Implement support for running jobs in a Singularity container

Use a find command to get the path to (JAR) files to avoid hard-coding version numbers into more than one place.

Redirect stderr to stdout (2>&1) when displaying version information for installed programs. I have noticed that some programs write to stdout and some programs write to stderr and it may even be different between two versions of the same program. To get a nice display the output should go to stdout.

[Nicklas Nordborg]

In 6686:

References #1374: Implement support for running jobs in a Singularity container

Added hostname to all containers.

[Nicklas Nordborg]

In 6692:

References #1374: Implement support for running jobs in a Singularity container

New version of opengrid.jar (1.5).

[Nicklas Nordborg]

In 6693:

References #1374: Implement support for running jobs in a Singularity container

Added global_env to reggie-config.xml and moved some configuration settings into export XXX statments there instead:

• <reference-folder> --> ReferenceFolder
• <container-folder> --> ContainerFolder
• <import-gateway> --> ImportGateway
• <import-archive> --> ImportArchiveRoot

[Nicklas Nordborg]

In 6694:

References #1374: Implement support for running jobs in a Singularity container

Fixed some incorrect path in configuration file.

[Nicklas Nordborg]

In 6695:

References #1374: Implement support for running jobs in a Singularity container

Only output messages if debugging.