1623329 : NAF jobs stay in idle state

Created: 2026-08-07T09:45:29Z - current status: new

Anonymized Summary:

Issue: User reports that their submitted jobs remain in the "idle" state for extended periods (overnight), despite previously starting within ~30 minutes. The jobs are lightweight (1 core, 512 MB–2 GB RAM) and were working correctly before the user’s recent absence.

Key Observations: - condor_q -analyze shows available slots (9 ready, 133 drainable), yet jobs fail to start. - Logs indicate repeated disconnections (Socket between submit and execute hosts closed unexpectedly) and failures to reconnect (Job not found at execution machine), leading to job evictions (Code 1008). - Jobs eventually get rescheduled but continue failing similarly across multiple nodes (e.g., slot2_X@batchNNN.desy.de).

Possible Root Cause: Network instability or misconfiguration causing socket closures between submit/execute hosts, preventing job execution despite resource availability.

Solution/Next Steps:

  1. Check Network Connectivity: Investigate potential firewall/network issues between the submission host and worker nodes (e.g., bird-htc-sched24.desy.de). Ensure ports/sockets used by HTCondor are stable.

  2. Review Node Health: Verify the status of affected batch nodes (e.g., batch1656, batch1557). Nodes may be dropping connections due to load or configuration errors.

  3. Test Simplified Submission: Submit a minimal test job (e.g., sleep 60) via condor_submit CLI to isolate whether the issue lies with the Python bindings or environment.

  4. Monitor Drain Status: Confirm if drainable slots (133 slots would match if drained) are stuck in draining mode, delaying job starts.

  5. Contact Support: Escalate to NAF/HTCondor admins with logs and timestamps to correlate with system-wide events during the user’s absence.