mamba environments¶

Lets assume you have successfully initialized your mamba setup, either through a mamba init section in your lgion script, or through

module load maxwell mamba
. mamba-init

You can then start to work with mamba environments.

Where to store mamba environments¶

GPFS is good location to store mamba environments. There are various possible locations:

The home directory: $HOME is an excellent place, and the default location when working with mamba. It does not require any special setup and is secured through snapshots. The space is however limited to 30GB, and mamba projects can quickly consume all your quota.
Recommendation: use your home-directory if you plan to create only a small number of not very complex environments.
DUST space: /data/dust/user/$USER is very well suitable. If you don't have such a folder, please drop a message to maxwell.service@desy.de. The default quota is 250GB which is usually sufficient for a decent number of fat environments. It doesn't have any security mechanisms, and it requires some configuration, but that's simple enough. In the following instructions we will assume DUST to be the storage location.
Recommendation: use DUST - unless you're sure that the 30GB quota of your home-directory is ample of space.
Group or instrument specific space: CFEL, CSSB, Eu.XFEL, Petra III/FLASH all have dedicated GPFS space.
Recommendation: if such space is available for you/your group/your instrument, that's the space you should use

There are various ways to specify your preferred location of mamba-related files. Defining it in ~/.condarc is preferable. Your Your ~/.condarc might look like this (replace with your username):

auto_activate_base: false

pkgs_dirs:
  - /data/dust/user/<user>/.conda/pkgs

envs_dirs:
  - /data/dust/user/<user>/.conda/envs

All mamba/conda environments and packages will then be stored in /data/dust/user/<user>/.conda/.

mamba/conda uses lots of hardlinks, and can refer to packages in various environments. If you have to work with environments in different locations, for example your private environments in DUST, and environments in /asapa3/ for beamline related tasks, it might be a good idea to separate the mamba location including the packages. To do so you could create a seconds condarc-file, e.g. /asap3/petra3/gpfs/common/p06/.condarc and use it whenever you work on these environments:

export CONDARC=/asap3/petra3/gpfs/common/p06/.condarc

Managing mamba environments¶

Mamba allows to create, clone, export, list, remove, and update environments with specific versions of Python and packages.

Creating environments¶

Open a fresh terminal, and initialized mamba if it's not part of your login anyway.

To create an environment:
```
mamba create --name <my-env>
```
Replace <my-env> with the name of your environment. 2. When mamba asks you to proceed, type y: That will create a folder in the conda-env directory, name ~/.conda/envs by default or in the directory specified by the envs_dirs configuration option. 3. To create an environment with a specific version of Python:
```
mamba create --name <my-env> python=3.9
```
To create an environment with a specific version of Python and multiple packages:
```
mamba create -n myenv python=3.9 scipy=0.17.3 scikit-image
```

Note

specifying all packages needed in an environment when creating the environment can help to avoid conflicts between package versions.

Creating environments with yaml files¶

Use the terminal for the following steps:

Create the environment from the environment.yml file:
```
mamba env create -f environment.yml
```
The first line of the yml file sets the new environment's name.
Activate the new environment:
```
mamba activate <my-env>
```
Verify that the new environment was installed correctly:
```
mamba env list
```
You can also use mamba info --envs

Creating a yaml file from an environment¶

You may want to share your environment with someone else - for example, so they can re-create a test that you have done. To allow them to quickly reproduce your environment, with all of its packages and versions, give them a copy of your environment.yml file.

Activate the environment to export:
```
mamba activate myenv
```
Export your active environment to a new file:
```
mamba env export > environment.yml
```
An existing environment.yml will be overwritten. The yaml-file contains both pip and conda packages. 3. Share the exported yaml file for example via gitlab.desy.de, which has several advantages, in particular allows easily re-creating an environment if it got corrupted.

If you want to make your environment file work across platforms, you can use the mamba env export --from-history flag. This will only include packages that you’ve explicitly asked for, as opposed to including every package in your environment. For example, if you create an environment and install Python and a package:

mamba install python=3.7 scipy

This will download and install numerous additional packages to solve for dependencies. This will introduce packages that may not be compatible across platforms. If you use mamba env export, it will export all of those packages. However, if you use mamba env export --from-history, it will only export those you specifically selected, which also make the yaml files much more concise. It will however not allow to create an exact copy of the environment. Versions of required packages can be different if a package like scipy did not require an exact version of numpy.

Installing environments in arbitrary locations¶

You can control where a conda environment lives by providing a path to a target directory when creating the environment. For example, the following command will create a new environment in /asap3/petra3/gpfs/common/p11/conda:

mamba create --prefix  /asap3/petra3/gpfs/common/p11/conda python=3.11 matplotlib scipy ...

You activate the environment also with the full path:

mamba activate /asap3/petra3/gpfs/common/p11/conda

Cloning environments¶

You can make an exact copy of an environment by creating a clone of it:

mamba create --name my-clone --clone my-env

and of course replace my-clone with the name of the new environment and my-env with the name of the existing environment to be cloned.

If you created an environment using a prefix, it works pretty much the same way:

mamba create --prefix=/path/to/my-clone --clone /path/to/my-env

Information about environments¶

By default, the active environment---the one you are currently using---is shown in parentheses () or brackets [] at the beginning of your command prompt:

(myenv) $

If you do not see this, run:

mamba info --envs  # OR
mamba env list

In the environments list that displays, your current environment is highlighted with an asterisk (*). That is however only the case, if the environment is listed in ~/.conda/environments.txt. If you activated for example an environment (possibly created by someone else) using the prefix, the environment won't show up. Simply use

which python

To see a list of all packages installed in a specific environment:

If the environment is not activated, in your terminal window, run:
```
mamba list -n my-env
```
If the environment is activated, in your terminal window, run:
```
mamba list
```
To see if a specific package is installed in an environment, in your terminal window, run:
```
mamba list -n myenv scipy
```

Using pip in environments¶

To use pip in your environment, you need to have it installed:

mamba install -n myenv pip
mamba activate myenv
python3 -m pip install <list of packages>

Issues may arise when using pip and conda together. When combining conda and pip, it is best to use an isolated conda environment. It's usually recommended not to switch back and forth between conda and pip, but rather start with conda and use pip only at the end. pip and conda however usually go well along.

Setting environment variables¶

If you want to associate environment variables with an environment, you can use the config API. This is recommended as an alternative to using activate and deactivate scripts since those are an execution of arbitrary code that may not be safe.

First, create your environment and activate it:

mamba create -n my-env
mamba activate my-env

To list any variables you may have, run conda env config vars list.

To set environment variables, run mamba env config vars set my_var=value.

Once you have set an environment variable, you have to reactivate your environment: mamba activate test-env.

To check if the environment variable has been set, run echo $my_var or mamba env config vars list.

You can specify the environment you want to affect using the -n and --prefix flags.

To unset the environment variable, run mamba env config vars unset my_var -n test-env.

Removing environments¶

To remove an environment run:

mamba remove --name my-env --all   # OR
mamba env remove --name myenv

To verify that the environment was removed:

mamba info --envs

Note: this page made heavy use of the conda documentation: https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html. For more details and additional features and commands please have a look at the conda documentation.

Staging environments to local disk¶

there are cases, where it's beneficial to run mamba environments from a local disk, for example to speed up calculations or for reliable benchmarks. Building conda environments for individual batch nodes (on the batch nodes local disk) is terribly inconvenient and inefficient. If you ever have a need for local installations, have a look at conda-stage which nicely takes care of such tasks.