#1374 closed task (fixed)
Implement support for running jobs in a Singularity container
Reported by: | Nicklas Nordborg | Owned by: | Nicklas Nordborg |
---|---|---|---|
Priority: | major | Milestone: | Reggie v4.38 |
Component: | net.sf.basedb.reggie | Keywords: | |
Cc: |
Description (last modified by )
In a Singularity container we can install all programs that are needed for a specific analysis step without having to worry about exactly which versions that are installed on the host system. In the simplest form a script can be started with:
singularity exec /path/to/container.sif job.sh
but we will at least also need to specify --bind
options so that the container can access project-archive
, run-archive
, and other directories where we store data.
We also need to split the job.sh
script, since we need the options for qsub
. For example:
start_job.sh is submitted to the queue system and contains the options is responsible for starting the job.sh
script:
# --- start of qsub options --- #$ -S /bin/sh #$ -terse .... # --- end of qsub options --- singularity exec /path/to/container.sif job.sh
Also, since the container typically know and have control over the paths were the programs are installed we should not have to specify them in the configuration file.
Note that we still want to support the non-container way of running jobs.
Here is the final solution
The submission procedure has changed quite a lot and there are now several scripts. Scripts that are dynamically generated are designed to be as small as possible and should in most cases only define environment variables and then call a static script that performs the actual work.
submit.sh
: This is a generated script and contain options for the job scheduler. It will start therun.sh
script.run.sh
: This is a static script that is mostly about error handling and is slightly different depending on the cluster type. It will start thejob.sh
script.job.sh
: This is a generated script with all options for running the selected analysis. It will setup paths to data folders, program parameters, etc. and then run another static script that is doing the actual analysis.
The job.sh
script is generated by a combination of settings in reggie-config.xml
and information that is stored in the database. There are major changes in the reggie-config.xml
, but all analysis steps are now very similar in how they are configured:
<global-env>
: Intended for setting up paths to common folders. For example the<reference-folder>
is now defined here asexport ReferenceFolder=...
. This setting is included injob.sh
for all analysis steps.
<..>/<env>
: section that is intended for setting up environment variables for a specific analsysis. Lots of settings that were configured by subtags show now go in this sections instead.<..>/<env-debug>
: this section is included if the job is running in debug mode.<..>/<execute>
: the command for executing the analysis script. This is typically as simple as./run-analysis.sh
but can be more complex if the analsyis should run in a container.
Here is an example for the StringTie step:
<stringtie> <env> export STRINGTIE="/usr/local/packages/stringtie/1.3.3b/bin/stringtie" export PREPDE="/usr/local/packages/stringtie/1.3.3b/bin/prepDE.py" export GTF="${ReferenceFolder}/hg38/hg38.analysisSet_gencode27_snp150/gencode.v27.primary_assembly.annotation_subset_transcripttype_proteincoding.gtf" export StringTieOptions="--rf -B -e" </env> <env-debug></env-debug> <execute>./stringtie.sh</execute> </stringtie>
When running in a container we do not need to specify paths to programs (since the container already knows this):
<stringtie> <env> export GTF="${ReferenceFolder}/hg38/hg38.analysisSet_gencode27_snp150/gencode.v27.primary_assembly.annotation_subset_transcripttype_proteincoding.gtf" export StringTieOptions="--rf -B -e" </env> <env-debug></env-debug> <execute> singularity exec --bind ${ArchiveFolder},${ReferenceFolder} ${ContainerFolder}/stringtie-v1-stringtie1.3.3b.sif ./stringtie.sh </execute> </stringtie>
Change History (34)
comment:1 by , 3 years ago
comment:32 by , 2 years ago
Description: | modified (diff) |
---|
comment:33 by , 2 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
In 6626: