Python¶

there are various python3 variants available on Maxwell. Python2 is however not available. The installations vary a lot in pre-installed modules. See the list of python modules for more details.

System python¶

EL9 comes with python 3.9.x. Without loading any modules, that is the version you will get by default.

@max-display004:~/Desktop$ python --version
   Python 3.9.18

@max-display004:~/Desktop$ python3 --version
   Python 3.9.18

Only a very basic set of python modules is installed for this python version. The main advantage of this python version is the consistency with libraries coming with EL9, in particular HDF5 and OpenMPI or other MPI variants. The python version is very well suited to create your own virtual environments. See working with virtual environments for instructions.

If you're happy with python 3.9 and virtual environments, use this version.

Vanilla python¶

Vanilla python installations are not very different from the system python, but available in a few more versions. Currently the versions are python 3.9, 3.10, 3.12.

The different python versions can be selected using modules

module load maxwell python       # enables python 3.12
module load maxwell python/3.9   # enables python 3.9

# to show avail python version:
module avail python

The build process is most simple and you can easily install your own variant if you should ever need one:

#!/bin/bash
version=3.12.3 # or whatever version you'd like to install
major=$(echo $version |  cut -d. -f1-2)
sdir=<where-ever-you-want-to-keep-it>/python/src
bdir=/scratch/$USER
idir=<your-choice-of-install-dir>
mkdir -p $sdir $bdir

if [[ ! -f  $sdir/Python-$version.tgz ]]; then
   wget https://www.python.org/ftp/python/$version/Python-$version.tgz -O $sdir/Python-$version.tgz
fi

cd $bdir
tar xf  $sdir/Python-$version.tgz
cd Python-$version
./configure --prefix=/software/python/$major
make -j 8
make install
export PYTHONNOUSERSITE=1
cd $bdir

# adjust for your set of modules
modules="numpy scipy matplotlib scikit-image scikit-learn xarray urllib3 SQLAlchemy Spyder tifffile sniffio requests PyQt5 PyMca5 pyFAI protobuf plotly pillow numba jupyter jupyterlab Jinja2 imageio h5py fabio Cython babel pandas"

/software/python/$major/bin/python3 -m pip install $modules

Vanilla pythons all come with venv and virtualenv, so are also quite suited to work with python virtual environments.

Conda python¶

conda is a quite versatile and popular module/environment management framework. It can work with any python3 version, and is quite good in keeping environments consistent. However, before using conda or mamba please have a look at the page sorting out conda, mamba, anaconda, in particular if you plan to install your own conda/mamba framework.

As mentioned on that page, conda and mamba are pretty much the same thing. We recommend using mamba, it's simply faster, but otherwise can be used interchangeably.

You can currently find two different mamba installations on the cluster:

mamba/3.9 That's a fat environment for python 3.9 and largely identical to the one available for Centos 7, which was also serving the jupyterHub
mamba/3.1 is a much more light-weight environment for python 3.11

You can use modules to select one or the other installation

module load maxwell mamba        # enables python 3.11
module load maxwell mamba/3.9    # enables python 3.9

# to show avail python version:
module avail mamba

Just for consistency, exactly the same modules are available for conda, and they do exactly the same as the mamba modules:

module load maxwell conda        # enables python 3.11
module load maxwell conda/3.9    # enables python 3.9

# to show avail python version:
module avail conda

You can finally modify your login using mamba init, but it can have some downsides. See working with mamba environments for instructions.

Common pitfalls¶

it's important to remember that python searches in a number of directories in a specific order. For example

export PYTHONPATH=/scratch
python3 -c 'import sys; print(sys.path)'

    '', '/scratch', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '~/.local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']

shows the order in which the different locations are visited. So python will look into

'': the current directory
/scratch: the PYTHONPATH
~/.local/lib/python3.9/site-packages: your personal site-packages folder
/usr/lib64/python3.9/site-packages: the systems site-packages folder

and will pick the first match.

Installing for example python modules with pip using

python3.9 -m pip install module --user

will install into ~/.local/lib/python3.9/site-packages, and that takes always precendence over packages installed in a virtual environment, which can lead to a rather confusing behavior and unexpected version mismatches.

So we recommend to avoid installing into ~/.local/lib/ and rather always use virtual/conda environments (which could also be a mamba base environment).