id summary reporter owner description type status priority milestone component resolution keywords cc 682 Improve performance when submitting jobs to cluster Nicklas Nordborg Nicklas Nordborg "Submitting multiple jobs to the cluster can take a long time. For example, submitting align jobs for an entire plate may take several minutes. After some initial investigations it seems like the minimal time it takes to submit a single command via SSH to the cluster is around 0.25 seconds. One align job requires 3 commands and one file transfer so it takes at least one second per job submitted. When submitting several jobs, there are things that can be improved: * Two commands are used to make sure that the job working directory is empty and exists: `rm -rf ` and `mkdir `. When submitting multiple jobs we could do all of this in a single command. * Sending the job scripts opens a new `SCPFileTransfer` every time. I have not checked if it is possible to re-use the same instance for transferring multiple files and if that is faster. * Registering jobs on the cluster should also be possible to do in a batch. It should just be a matter of keeping track of the order in the output file so we assign the correct job id to each job. " enhancement closed major Reggie v2.17 net.sf.basedb.reggie fixed