Changes that are needed or useful for running jobs inside a container

[Nicklas Nordborg]

I expect that we want to add some functionality and change some things to make it easier to implement support for running jobs inside a container.

Some things that have been discovered so far:

• The options that causes set -e and umask to be included are better handled in the actual scripts. The JobConfig.setFailImmediately() and JobConfig.setCreatePrivateFiles() should be deprecated and not used.
• The job.sh scripts need to export variables (eg. TMPDIR and WD otherwise they will not be available in scripts that are started as sub-scripts.
• more...

[Nicklas Nordborg]

In 6623:

References #1375: Changes that are needed or useful for running jobs inside a container

Exporting variables that can be used by scripts.

[Nicklas Nordborg]

In 6624:

References #1375: Changes that are needed or useful for running jobs inside a container

Deprecated and removed functionality related to JobConfig.setFailImmediately() and JobConfig.setCreatePrivateFiles().

[Nicklas Nordborg]

In 6625:

References #1375: Changes that are needed or useful for running jobs inside a container

Added ScriptBuilder.export() and ScriptBuilder.var() methods.

[Nicklas Nordborg]

In 6627:

References #1375: Changes that are needed or useful for running jobs inside a container

Automatically set execute permission on files that ends with '.sh'.

Added ScriptBuilder.toUploadSource() method.

[Nicklas Nordborg]

In 6629:

References #1375: Changes that are needed or useful for running jobs inside a container

Changed all scripts to use '/bin/bash' instead of '/bin/sh' since my test installation with Ubuntu defaults to 'dash' which casused problems with some scripts.

The OpenGrid and Slurm engines now split some functionality to static scripts.

[Nicklas Nordborg]

In 6634:

References #1375: Changes that are needed or useful for running jobs inside a container

Clusters are listed in the same order as they are defined in the configuration file.

[Nicklas Nordborg]

In 6639:

References #1375: Changes that are needed or useful for running jobs inside a container

Added a 'nice' value to direct jobs since that will work better with virtual box running with a single cpu.

When updating the status file for the first time it should be with the '>' operator and not the '>>' since the latter may just append to an existing file that is not used any longer.

[Nicklas Nordborg]

In 6648:

References #1375: Changes that are needed or useful for running jobs inside a container

Added a method for joining one script to another.

[Nicklas Nordborg]

In 6659:

References #1375: Changes that are needed or useful for running jobs inside a container

Added OpenGridSession.executeNow() method which executes a command and has support for uploading script files before the command is executed.

[Nicklas Nordborg]

In 6660:

References #1375: Changes that are needed or useful for running jobs inside a container

Names for jobs are generated based on date/time and a counter instead of a random GUID. This makes it easier when debugging since the names (=directories) are sorted in the same order as jobs are submitted.

[Nicklas Nordborg]

In 6666:

References #1375: Changes that are needed or useful for running jobs inside a container

Add more file extensions that should be "executable" if no explicit permission has been set: .sh, .pl, .awk, .py, .R

[Nicklas Nordborg]

In 6667:

References #1375: Changes that are needed or useful for running jobs inside a container

Also add variables to the job.sh script so that it should be possible to run it for debugging.

Do not clear an existing temporary directory when debugging.

[Nicklas Nordborg]

In 6668:

References #1375: Changes that are needed or useful for running jobs inside a container

Append to the stderr file instead of overwrite if the user aborts a script the is running with debugging.