1589990 : Unable to access NAF Jupyter lab

Created: 2026-04-07T10:11:14Z - current status: new

Anonymized Summary: A user reports being unable to access their JupyterLab instance on the National Analysis Facility (NAF) platform. The error encountered is a "504 Gateway Time-out" when attempting to connect to the notebook URL (e.g., https://naf-jhub.desy.de/user/[USERNAME]).

Possible Causes & Solutions:

  1. Deadlocked Notebook Entry in the Hub
  2. The user’s notebook may be stuck in a deadlocked state, preventing normal shutdown or reconnection. This can occur due to:
    • A failed route deletion in the proxy (e.g., timeout exceptions).
    • A race condition where the HTCondor scheduler removes a notebook while the slot is quickly reassigned.

  3. Solution:

    • Try logging out and back in to force a fresh session.
    • If the issue persists, the user may need to wait for the hub’s periodic cleanup (or contact admins if the patch for self-removal of deadlocked entries is not yet active).
    • Check the .jupyterhub.condor.err and .jupyterhub.condor.out logs in the user’s $HOME directory for errors.

  4. Unresponsive CVMFS Mounts

  5. Hanging CVMFS mounts on the worker node may prevent the notebook from reporting its port back to the hub, leading to a timeout.

  6. Solution:

    • Retry spawning the notebook after some time (faulty nodes should be removed by the healthcheck system).
    • If the issue recurs, check the logs for CVMFS-related errors.

  7. Disk Quota Exceeded

  8. The error log snippet (Disk quota exceeded) suggests the user’s $HOME directory may be full, blocking critical operations.

  9. Solution:

    • Clean up unnecessary files in $HOME or request a quota increase.
    • Verify disk usage with fs lq (AFS) or quota commands.

  10. Jupyter Configuration Issue

  11. A misconfigured local Jupyter setting (e.g., open_browser = False in .jupyter/jupyter_server_config.json) may interfere with the hub connection.

  12. Solution:

    • Comment out or remove the open_browser entry in the config file.

  13. General Timeout Causes

  14. The hub may be under heavy load, or the worker node may be unresponsive.
  15. Solution:
    • Wait a few minutes and retry.
    • Check the NAF status page or contact support if the issue persists.
  1. Check Logs:
  2. Review .jupyterhub.condor.err and .jupyterhub.condor.out in $HOME for specific errors.
  3. Free Up Disk Space:
  4. Run fs lq to check quota usage and delete unnecessary files.
  5. Retry Spawning:
  6. Log out, clear browser cache, and attempt to reconnect.
  7. Contact Support:
  8. If the problem remains unresolved, reach out to the NAF support team with the error logs.

Sources Used: - NAF Documentation: Notebook Spawning Issues - NAF Documentation: JupyterHub Logs