| | 53 | In it's simplest form a job script is only a string with one or more (bash) commands to execute. For example, `pwd; ls` is a valid job script that will print the current directory and then list all files in it. To help creating longer and more complex scripts the `ScriptBuilder` class can be used. The `cmd()`, `echo()` and `comment()` methods are more or less self-describing. It is possible to start a command in the background with `bkgr()`, but not that this must be paired with a `waitForProcess()` otherwise the job script may finish before the commmand that is running in the background which may cause unpredictable results. The `progress()` method is a very useful method for jobs that are expected to take a long time to run. The method will write progress information to the `{$WD}/progress` file which will be picked up by the Open Grid Service and reported back to the BASE job that is acting as a proxy. |
| | 54 | |
| | 55 | When creating a job script there are a few useful variables that has been set up: |
| | 56 | |
| | 57 | * `{$WD}`: A randomly generated subdirectory in the `<job-folder>` directory. The directory contains the job script which is also the current working directory when the job is started and the directory that is used for communicating data to/from the BASE server. Data in this directory is preserved after the job has finished. When running post-job code this folder can be found by calling `OpenGridCluster.getWorkFolder()`. Files can be downloaded to the BASE server with `OpenGridSession.downloadFile()`, `OpenGridSession.readFile()` or `OpenGridSession.getJobFileAsString()`. The latter method is the simplest one to use for parsing out interesting data from text result files. |
| | 58 | * `{$TMPDIR}`: A temporary working directory that is typically only available on the node the job is running on. Unless the job is started in debug mode, this directory is deleted soon after the has been completed. |
| | 59 | * `{NSLOTS}`: The number of slots that has been assigned to this job. If the job is starting a multi-threaded analysis program it is common practice to not use more threads that what this value specifies. Note that a single node may run more than one job at the same time so using `nproc` to determine the number of threads may cause resource issues. |
| | 60 | |
| | 61 | In the example code below we assume that we have FASTQ files stored on a file server on the network. The FASTQ files are going to be aligned with Tophat and we have a wrapper script that sets all parameters except the number of threads and the location of the FASTQ files. After Tophat we have a second post-alignment script that does some stuff and save the result in a subdirectory. |
| | 62 | |
| | 63 | {{{ |
| | 64 | ScriptBuilder script = new ScriptBuilder(); |
| | 65 | // We do not want to hog the network so we copy all files we need to the local cluster node |
| | 66 | script.progress(5, "Copying data to temporary folder..."); |
| | 67 | script.cmd("cp /path/to/fastqfiles/*fastq.gz {$TMPDIR}"); |
| | 68 | |
| | 69 | // Wrapper script that calls tophat; we assume all other required parameters are set by the wrapper |
| | 70 | script.progress(10, "Analysing FASTQ files with Tophat..."); |
| | 71 | script.cmd("tophat-wrapper.sh -p {$NSLOTS} {$TMPDIR}"); |
| | 72 | |
| | 73 | // Another analysis script... |
| | 74 | script.progress(50, "Post-alignment analysis files..."); |
| | 75 | script.cmd("post-analysis.sh -p {$NSLOTS} {$TMPDIR}"); |
| | 76 | |
| | 77 | // Now we only need to copy the results back to our file server. |
| | 78 | // Remember that the {$TMPDIR} is cleaned automatically so we don't have to mess with that |
| | 79 | script.progress(90, "Copying analyzed data back to file server"); |
| | 80 | script.cmd("cp {$TMPDIR}/result/* /path/to/resultfiles/"); |
| | 81 | |
| | 82 | // Finally, we copy the logfile to the job directory so that we can extract data from it to BASE |
| | 83 | script.cmd("cp {$TMPDIR}/logfile {$WD}/logfile"); |
| | 84 | }}} |
| | 85 | |
