pixi¶

Summary

Source: https://github.com/prefix-dev/pixi/

License: BSD 3-Clause "New" or "Revised" License

Path: /software/pixi

Documentation: https://pixi.sh/latest/

Pixi is a fast, modern, and reproducible package management tool for developers of all backgrounds.

pixi is an excellent python package manager with unique features in particular when deploying python environments in a highly reproducible manner.

Please consult the exhaustive pixi documentation for details. We provide just a few examples below.

Using pixi¶

we provide a pixi installation which can be initialized using modules

module load maxwell pixi
pixi

You might consider using a private pixi installation, and set PIXI_HOME to a suitable location like /data/dust/user/$USER/pixi. The installation is lightweight, but can become quite large when creating various environments.

pixi environments¶

pixi environments are slightly different from conda environments. First of all there is no base environment and no conda init, which tends to confusticate the environment. It's however possible to "convert" a pixi environment into a portable conda-like environment (see below).

Creating pixi environments¶

Creating a pixi environment is as simple as create any other python virtual environment:

# setup
module load maxwell pixi

# create an environment
pixi init /data/dust/user/$USER/TF   # --format pyproject

That just defines the layout of the environment but doesn't actually contain any python interpreter or packages. To add packages you essentially have to options:

Use pixi shell¶

pixi shell has the advantage that you can simply deactivate bey exiting the shell, but you have to cd into the environment

export PIXI_HOME=/data/dust/user/$USER/pixi
cd /data/dust/user/$USER/TF
pixi add python=3.12 tensorflow tensorboard
pixi shell # OR
cd $HOME
pixi_activate /data/dust/user/$USER/TF

Use pixi_activate¶

pixi_activate will setup the environment and add the moxt recent python interpreter. There is no deactivate, start a fresh shell instead. It works from any location, you don't have to cd into the environment.

cd $HOME
pixi_activate /data/dust/user/$USER/TF
# downgrade python and add packages
pixi add python=3.12 tensorflow tensorboard

pixi_activate is quite handy when using the environment.

Note: don't name the environment like a python package you intend to install. It tends to confuse pixi.

Information about the setup¶

# get information about the setup
pixi info

# get information about packages in an environment
pixi_activate  /data/dust/user/$USER/TF
pixi list

# get help
pixi help

Using the environment without activation¶

it's sufficient to simply set the PATH:

export PATH=/data/dust/user/$USER/TF/.pixi/envs/default/bin:$PATH

pixi has many more nice features. Please consult the manuals!

Packing pixi environments¶

One of the great advantages of pixi is the ability to create relocatable packages. A simple sample:

# setup
module load maxwell pixi
cd /data/dust/user/$USER

# create env
pixi init pixiTF
cd pixiTF
pixi add python=3.11 pip tensorflow tensorboard

# packaging can require quite a bit of tmp and cache-space
export PIXI_CACHE_DIR=/data/dust/user/$USER/.cache
export TMPDIR=/data/dust/user/$USER/.tmp
mkdir -p $PIXI_CACHE $TMPDIR

# create the package (tarball)
pixi-pack  -o ../tensorflow.tar --environment default --platform linux-64 pixi.toml

Unpacking and deploying pixi environments¶

the tarball created above can be hosted anywhere. pixi-unpack doesn't seem to support URLs for unpacking from remote sources, but that's a minor detail:

# setup
module load maxwell pixi

# fetch and unpack in an arbitrary location
cd /data/dust/user/$USER/TEST
wget -q https://docs.desy.de/samples/pixi/tensorflow.tar && pixi-unpack tensorflow.tar

# actitvate
source activate.sh

# check
python3 --version
    # Python 3.11.13

python3 -c 'import tensorflow as tf'
    # works

Using self-extracting binaries¶

pixi environment can be deployed even on systems without any pixi installation. pixi self-extracting executable bundle pixi-unpack together with the environment.

Create the environment:

# setup
module load maxwell pixi
cd /data/dust/user/$USER

# create env
pixi init pixiTF
cd pixiTF
pixi add python=3.11 pip tensorflow tensorboard

# packaging can require quite a bit of tmp and cache-space
export PIXI_CACHE_DIR=/data/dust/user/$USER/.cache
export TMPDIR=/data/dust/user/$USER/.tmp
mkdir -p $PIXI_CACHE $TMPDIR

# create the package (tarball)
pixi-pack  -o ../tensorflow-install.sh --create-executable

Put the installer to a downloadable location and deploy the environment elsewhere:

# unpacking can require quite a bit of tmp and cache-space
export PIXI_CACHE_DIR=/data/dust/user/$USER/.cache
export TMPDIR=/data/dust/user/$USER/.tmp
mkdir -p $PIXI_CACHE $TMPDIR

wget https://docs.desy.de/samples/pixi/tensorflow-install.sh && /bin/sh tensorflow-install.sh
source activate.sh

# check if tensorflow is installed and try it out
python3 -m pip list 
python3 -c 'import tensorflow as tf'

Using a container registry¶

container registries are awesome for managing and deploying container images. To use the registry to manage self-extracting binaries you need to wrap the binaries into a lightweigt container.

Using container to deploy self-extracting binaries¶

Create a Dockerfile in the directory containing the self-extracting binary:

FROM alpine:latest
RUN apk update && apk add bash
COPY tensorflow-install.sh /usr/local/bin/installer.sh
RUN chmod +x /usr/local/bin/installer.sh
ENTRYPOINT ["/usr/local/bin/installer.sh"]

Build and upload the image

podman build -t tensorflow-installer:20250718 .
podman tag localhost/tensorflow-installer:20250718 tollerort.desy.de/maxsoft/tensorflow-installer:20250718
podman push tollerort.desy.de/maxsoft/tensorflow-installer:20250718

Use the image to install the pixi environment elsewhere

# just run the image with apptainer, singularity or podman
singularity run docker://tollerort.desy.de/maxsoft/tensorflow-installer:20250718

source activate.sh 
which python
   # /home/schluenz/TEST/env/bin/python
python -c 'import tensorflow as tf'
   # works

Wrapping the installer into images makes the integration into CI/CD pipeline quite simple.

Noteworthy¶

pixi-pack/unpack won't work for everything. It can handle PyPi wheel packages, but not source distributions.
cuda dependencies might not be fully portable.
packages can become quickly quite fat. Make sure to have sufficient space in PIXI_HOME, PIXI_CACHE_DIR and TMPDIR.