1600507 : JupyterHub Spawn failed

Created: 2026-05-12T15:08:56Z - current status: new

Anonymized Summary:

A user reports persistent issues when attempting to start a JupyterHub notebook server. The error message indicates a timeout during the connection phase:

"Spawn failed: Server at http://[WORKER_NODE]:[PORT]/user/[USERNAME]/api didn't respond in 60 seconds."

The user has encountered this problem for several weeks.

Possible Causes & Solutions:

  1. Deadlocked Notebook Entry in Hub
  2. The user may have an obsolete server entry in the JupyterHub database that cannot be removed normally (e.g., due to a race condition or failed route deletion).

  3. Solution:

    • Try using the "Stop My Server" button in the JupyterHub GUI (if available). A patch (deployed in February 2026) allows users to force-remove deadlocked entries.
    • If the issue persists, contact NAF admins to manually remove the account or restart the hub.

  4. Disk Quota Exceeded

  5. The error log (.jupyterhub.condor.err in the user’s $HOME directory) may show: Failed to write server-info to [PATH]: OSError(122, 'Disk quota exceeded')

  6. Solution:

    • Check disk usage (quota -s or du -sh ~).
    • Clean up unnecessary files or request a quota increase.

  7. Unresponsive CVMFS Mounts

  8. Hanging CVMFS mounts (observed in December 2024) can prevent the notebook from reporting its port to the hub, leading to timeouts.

  9. Solution:

    • Retry spawning later (healthcheck bugs have been fixed, but mounts may still cause delays).
    • If the issue recurs, note the worker node (batchj003.desy.de in this case) and report it to admins.

  10. Local Jupyter Configuration Issue

  11. A misconfigured setting (e.g., open_browser = False in .jupyter/jupyter_server_config.json) may block the hub connection.

  12. Solution:

    • Check the file and comment out problematic entries.

  13. Worker Node Issues

  14. The assigned worker (batchj003.desy.de) might be overloaded or misconfigured.
  15. Solution:
    • Wait and retry later.
    • If the problem persists, report the node to admins.
  1. Check Error Logs:
  2. Inspect .jupyterhub.condor.err in the $HOME directory for specific errors (e.g., quota issues).
  3. Verify Disk Usage:
  4. Run quota -s to confirm no quota limits are exceeded.
  5. Force-Remove Deadlocked Server:
  6. Use the "Stop My Server" button in the JupyterHub GUI.
  7. Contact Admins if Needed:
  8. If the issue persists, provide the worker node name (batchj003.desy.de) and error logs to NAF support.

Sources: - NAF Documentation: Notebook Spawning Issues - JupyterHub BatchSpawner Issue #314