Opened 9 months ago

Closed 7 months ago

#1375 closed enhancement (fixed)

Changes that are needed or useful for running jobs inside a container

Reported by: Nicklas Nordborg Owned by:
Priority: major Milestone: Job scheduler extension v1.5
Component: net.sf.basedb.opengrid Keywords:
Cc:

Description

I expect that we want to add some functionality and change some things to make it easier to implement support for running jobs inside a container.

Some things that have been discovered so far:

  • The options that causes set -e and umask to be included are better handled in the actual scripts. The JobConfig.setFailImmediately() and JobConfig.setCreatePrivateFiles() should be deprecated and not used.
  • The job.sh scripts need to export variables (eg. TMPDIR and WD otherwise they will not be available in scripts that are started as sub-scripts.
  • more...

Change History (14)

comment:1 Changed 9 months ago by Nicklas Nordborg

In 6623:

References #1375: Changes that are needed or useful for running jobs inside a container

Exporting variables that can be used by scripts.

comment:2 Changed 9 months ago by Nicklas Nordborg

In 6624:

References #1375: Changes that are needed or useful for running jobs inside a container

Deprecated and removed functionality related to JobConfig.setFailImmediately() and JobConfig.setCreatePrivateFiles().

comment:3 Changed 9 months ago by Nicklas Nordborg

In 6625:

References #1375: Changes that are needed or useful for running jobs inside a container

Added ScriptBuilder.export() and ScriptBuilder.var() methods.

comment:4 Changed 9 months ago by Nicklas Nordborg

In 6627:

References #1375: Changes that are needed or useful for running jobs inside a container

Automatically set execute permission on files that ends with '.sh'.

Added ScriptBuilder.toUploadSource() method.

comment:5 Changed 9 months ago by Nicklas Nordborg

In 6629:

References #1375: Changes that are needed or useful for running jobs inside a container

Changed all scripts to use '/bin/bash' instead of '/bin/sh' since my test installation with Ubuntu defaults to 'dash' which casused problems with some scripts.

The OpenGrid and Slurm engines now split some functionality to static scripts.

comment:6 Changed 9 months ago by Nicklas Nordborg

In 6634:

References #1375: Changes that are needed or useful for running jobs inside a container

Clusters are listed in the same order as they are defined in the configuration file.

comment:7 Changed 9 months ago by Nicklas Nordborg

In 6639:

References #1375: Changes that are needed or useful for running jobs inside a container

Added a 'nice' value to direct jobs since that will work better with virtual box running with a single cpu.

When updating the status file for the first time it should be with the '>' operator and not the '>>' since the latter may just append to an existing file that is not used any longer.

comment:8 Changed 8 months ago by Nicklas Nordborg

In 6648:

References #1375: Changes that are needed or useful for running jobs inside a container

Added a method for joining one script to another.

comment:9 Changed 8 months ago by Nicklas Nordborg

In 6659:

References #1375: Changes that are needed or useful for running jobs inside a container

Added OpenGridSession.executeNow() method which executes a command and has support for uploading script files before the command is executed.

comment:10 Changed 8 months ago by Nicklas Nordborg

In 6660:

References #1375: Changes that are needed or useful for running jobs inside a container

Names for jobs are generated based on date/time and a counter instead of a random GUID. This makes it easier when debugging since the names (=directories) are sorted in the same order as jobs are submitted.

comment:11 Changed 8 months ago by Nicklas Nordborg

In 6666:

References #1375: Changes that are needed or useful for running jobs inside a container

Add more file extensions that should be "executable" if no explicit permission has been set: .sh, .pl, .awk, .py, .R

comment:12 Changed 8 months ago by Nicklas Nordborg

In 6667:

References #1375: Changes that are needed or useful for running jobs inside a container

Also add variables to the job.sh script so that it should be possible to run it for debugging.

Do not clear an existing temporary directory when debugging.

comment:13 Changed 8 months ago by Nicklas Nordborg

In 6668:

References #1375: Changes that are needed or useful for running jobs inside a container

Append to the stderr file instead of overwrite if the user aborts a script the is running with debugging.

comment:14 Changed 7 months ago by Nicklas Nordborg

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.