Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save OliPelz/0440c87c78636b75c836f7955e34d5da to your computer and use it in GitHub Desktop.
Save OliPelz/0440c87c78636b75c836f7955e34d5da to your computer and use it in GitHub Desktop.
How to user torque openpbs in python, a small howto
Just a small note, here is how you can convert commandline calls in python to use openpbs/torque
If the command is doing something in the background without any dependency to the main script thats easy,
just use qsub in your subprocess call and let if fork off.
But what you do if you have some subprocess calls which wait for a program or script to finish (it depends on it)?
After some Internet research I found qsub's parameter "-I -x" to be exactly what I needed!
lets say you have the following command fragment:
cmd = 'lastdb -Q 0 %s %s' % (last_index, validated_ref) # format the database
proc = subprocess.Popen(cmd, shell=True)
status = proc.wait()
Here is the solution how this can be written to use openpbs using its interactive mode (-I) but you need to take care that no interactive shell
will be openend (-x)
Here is how to replace the code:
scriptDir = os.path.dirname(os.path.realpath(__file__))
cmd = 'lastdb -Q 0 %s %s' % (last_index, validated_ref) # format the database
cmd = "qsub -I -x -v CMD='" + cmd + "' -N 'last_index_build' " + scriptDir + "/openpbs.sh"
proc = subprocess.Popen(cmd, shell=True)
status = proc.wait()
#PBS -S /bin/bash
#PBS -V
# put in here other stuff e.g. #PBS -l nodes=1:ppn=5
# put in here other stuff e.g. #PBS -l mem=4gb
$CMD 2> /dev/null
Here is an example I came across which includes some more advanced replacement technique:
If you have python code which runs a subprocess and pipes some data into it e.g.
read = '>%s \r\n%s' % (seqid, fastqhash['seq'])
cmd = 'bwa mem -x ont2d %s %s -' % (options,
ref_fasta_hash[dbname]['bwa_index'])
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE, shell=True)
(out, err) = proc.communicate(input=read)
status = proc.wait()
You need to work with files instead of pipes because I have not found an easy mechanism to pipe data into a qsub command script on the shell.
Discussions can be found here -http://stackoverflow.com/questions/18322286/how-can-i-use-a-pipe-or-redirect-in-a-qsub-command
But I think it is more easy to write the content into a temp file e.g.
bwaTmpDir = os.path.dirname(os.path.realpath(ref_fasta_hash[dbname]['bwa_index']))
f = tempfile.NamedTemporaryFile(dir=bwaTmpDir, delete=False)
f.write(read)
f.close()
scriptDir = os.path.dirname(os.path.realpath(__file__))
cmd = 'bwa mem -x ont2d %s %s %s' % (options,
ref_fasta_hash[dbname]['bwa_index'], f.name)
cmd = ["qsub", "-I", "-x", "-v", "CMD=" + cmd, "-N", "'run_bwa_align'", scriptDir + "/openpbs.sh"]
out = None
try:
out = subprocess.check_output(cmd)
except subprocess.CalledProcessError, e:
err = e.output
# There is a problem with this approach though, have you found out?
# there is no parameter to tell qsub to supress all internal output when using -I -x so the output
# will always contain qsub related stuff and unnecessary newlines, e.g.
qsub: job 777.xxx.xxx.de ready
qsub: job 777.xxx.xxx.de completed
Quick fix is to remove those qsub: lines and empty lines or write the output of the qsub command to a file, check out my class
pbs_utils.py which does all of that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment