More sophisticated Job submit strategies¶

It should always be the main goal to submit as many jobs as possible as part of one 'cluster' instead for ex. traversing through a directory in a 'foreach-loop' and invoke a 'condor_submit' command for every single find. Here are some more sophisticated examples on how to submit several jobs as one cluster of jobs which keeps the stress on the scheduler low and the responsitivity high ! If you use the techniques described below it NEVER hurts to use a 'circuit-breaker' option wit condor_submit: condor_submit -maxjobs # stop submitting after jobs

Last tip befor the actual submit file examples, you can use the 'dry' option to check what condor_Submit will actually do without submitting any job: condor_submit -dry - Submit file examples:

Go over a dir in 'foreach-style'¶

submit file:

( ... ))

Args = $(Item)

queue 1 Item matching files (*.dat)

In the submit dir:

ls *.dat 1.dat 2.dat 3.dat 4.dat

In the executable:

MY_INPUT_FILE = $1

In the queue:

[chbeyer@htc-it01]~/htcondor/testjobs% condor_submit sleep_foreach.submit Submitting job(s)....
4 job(s) submitted to cluster 2203711.

[chbeyer@htc-it01]~/htcondor/testjobs% condor_q -nobatch
-- Schedd: bird-htc-sched01.desy.de : <131.169.56.32:9618?... @ 06/29/18 12:05:07
 ID         OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
2203711.0   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 1.dat
2203711.1   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 2.dat
2203711.2   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 3.dat
2203711.3   chbeyer         6/29 12:04   0+00:00:00 I  0    0.0 sleep_args.sh 4.dat

Using a script for the input of arguments, each line of the scriptoutput is treated as an item¶

[chbeyer@htc-it01]~/htcondor/testjobs% cat items.sh 
#!/bin/bash
ls -1 *.dat

[chbeyer@htc-it01]~/htcondor/testjobs% ./items.sh 
1.dat
2.dat
3.dat
4.dat

The submit file:

Args = $(Item)

queue from ./items.sh |

Go through a list and reads the arguments from every line¶

[chbeyer@htc-it02]~/htcondor/testjobs% cat list.txt
60, eine
120, zwei
180, drei
240, vier

The submit file:

Args = $(var1) $(var2)

queue var1,var2 from list.txt

In the queue:

-- Schedd: bird-htc-sched02.desy.de : <131.169.56.95:9618?... @ 07/17/18 11:45:24
 ID         OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
6253267.0   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 60 eine
6253267.1   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 120 zwei
6253267.2   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 180 drei
6253267.3   chbeyer         7/17 11:45   0+00:00:00 I  0    0.0 sleep_args.sh 240 vier