Late materialization - a way around the job limit¶

The current maximum of jobs per user on any scheduler in the NAF is set to 5.000, if you want to submit more jobs e.g. before a holiday period or because you just can - use the late materialization features of condor:

The max_idle limit specifies the maximum number of non-running jobs that should be materialized in the condor_schedd at any one time. One or more jobs will materialize whenever a job enters the Run state and the number of non-running jobs that are still in the condor_schedd is less than this limit. This limit is stored in the Cluster ad in the JobMaterializeMaxIdle attribute.

The max_materialize limit specifies an overall limit on the number of jobs that can be materialized in the condor_schedd at any one time. One or more jobs will materialize when a job leaves the condor_schedd and the number of materialized jobs remaining is less than this limit. This limit is stored in the Cluster ad in the JobMaterializeLimit attribute.

Late materialization can be used as a way for a user to submit millions of jobs without hitting the MAX_JOBS_PER_OWNER or MAX_JOBS_PER_SUBMISSION limits in the condor_schedd, since the condor_schedd will enforce these limits by applying them to the max_materialize and max_idle values specified in the Cluster ad.

Examples:¶

`

submit as a factory with an idle jobs limit¶

max_idle = 100

submit 15,000 instances of this job¶

queue 15*1000`

Use condor_q -factory to query details about your submitted late-materialization job:

`

condor_q -factory -- Schedd: submit.example.org : <192.168.101.101:9618?... @ 12/01/20 13:35:00 ID OWNER SUBMITTED LIMIT PRESNT RUN IDLE HOLD NEXTID MODE DIGEST 77. bob 12/01 13:30 15000 130 30 80 20 1230 /var/lib/condor/spool/77/condor_submit.77.digest `