Skip to content

JupyterHub on Maxwell

A JupyterHub provides a multi-user environment to run notebooks slightly more conveniently. If you want to know more about JupyterHubs we'd recommend the official documentation.

The JupyterHub on the maxwell cluster and can be reached at https://max-jhub.desy.de and is available from outside the DESY Network. For login use your DESY credentials. This only works if you also have access to the Maxwell compute cluster. Using the Hub is not good enough a reason to grant you the access to the Maxwell cluster. In that case you can still use your single-user server and execute any notebook of your choice.

Configuration and Options

Let's assume that you successfully logged in https://max-jhub.desy.de. You will see a spawner field which allows you to choose between a few options:

Partitions

the Maxwell partitions selection allows to choose a partition or reservation.

  • the default is always the shared jhub partition. Each node is shared by up to 46 users!
  • the lifetime of the session is 7 days on the jhub partition, 8 hours on any other partition.
  • the lifetime for sessions using a reservation is set to the duration of the reservation.
  • not all existing partitions are shown. It should however only show partitions & reservations you're entitled to use.

Python

the Python Version selection allows to choose between different python environments. Currently available are

  • conda Python 3.11
  • conda Python 3.9

Use the default python 3.11 if you don't have a particular need for python 3.9.

Launch modus

the Launch modus selection allows to choose between different Jupyter environments:

  • Launch JupyterLab: launches a Jupyter Lab instance.
  • Launch Classical Notebook: the old-fashioned style, but launches much faster than lab notebooks.

Nodes

the Number of Nodes selection allows to set the node allocation. Nodes is always 1 for the jhub partition.

Constraints

The Constraints field allows to select nodes with specific features. Specify constraints in slurm style but without the --constraint= slurm directive, e.g. 'V100|A100' would be a valid constraint.

Environment variables

the Environment field allows to define environment variables, one or multiple entries per line, e.g.

  • CUDA_VISIBLE_DEVICE=0 OPENCL_VENDOR_PATH=/etc/OpenCL/vendors would be a valid setup.
  • Something like PATH=HOME/blubber:PATH should also work, but don't expect $HOME/blubber to be first in the PATH, the PATH to the python-interpreter (conda/mamba) and possibly others will always precede.
  • use JUPYTERLAB_DIR=~/.jupyter/lab to mimick the JupyterLab with user extensions option

Starting a Jupyter Server

  • The ? icon opens a small popup with some basic information. The popup persists until you click on the popup.
  • The table at the bottom shows the currently available resources for all partitions you are entitled to use.
  • There are several caches involved which can prevent an update of the list of partitions and reservation. Login, clear browser caches and login again if that appears to be a problem.
  • The table with node availability is generated in 2 minutes intervals. Reload the page to update the figures, but you will loose options specified.
To get really accurate information about node availability connect to max-display (ssh, fastx) and use savail 
savail -p allcpu,ps
savail gives a detailed status of nodes. The line with state available shows what really is available including nodes allocated to jobs which can be preempted.

Hit the spawn button to launch your notebook server.

Stopping a Jupyter Server

Jupyter notebooks are very inefficient way of using Maxwell resources. This is in particular true for abandoned and forgotten sessions. Please terminate your notebook server when you are done!

There are various ways to stop the notebook server ...

  1. Cancelling the slurm job:
    1. login to a maxwell node (e.g. max-display.desy.de),
    2. check if jobs are running (e.g. squeue -u $USER -a ) and
    3. terminate all your jupyter sessions with scancel -u $USER -n spawner-jupyterhub.
  2. Classical Notebook:
    1. use the Hub Control Panel button
    2. select Stop Server and logout
  3. JupyterLab:
    1. Under Files select the Hub Control Panel,
    2. select Stop Server and logout
  4. Visit https://max-jhub.desy.de/hub/home and stop your server.

Kernels

Jupyter kernels now reside in /software/jupyter/share/kernels/. Selection of kernels are identical for conda/3.11 and conda/3.9 environments. The jupyterhub will only see the latest version for a particular application like tensorflow, but you can easily enable any kernel if needed. The selection of kernels will roughly look like shown in the image. EuXFEL kernels will only show if you are a member of groups exfel or upex. JupyterLab has no way to organize kernels into categories or things alike. You can however whitelist the kernels you really need. See Configuring Jupyter Kernels for more details.
Please contact maxwell.service@desy.de if
  • you are missing kernels you'd find useful
  • available environments lack specific modules
  • you encounter problems (not everything is fully tested)
  • or for any other feedback ...