comsyl

Summary

Source: https://github.com/mark-glass/comsyl

License: MIT License

Path: /software/comsyl

Documentation: https://github.com/mark-glass/comsyl/wiki

Related: oasys srw

COMSYL makes the coherent mode decomposition of synchrotron radiation emitted by electrons passing through an undulator placed in a storage ring. COMSYL permits naturally the statistical analysis and propagation of the cross spectral density along the beamline optics. The coherence properties of the X-ray beam at any point of the beamline are completely given in terms of the eigenvalues and coherent modes of the cross spectral density (copied from https://github.com/mark-glass/comsyl ).

Using comsyl

comsyl is available as a standalone installation, compiled with openmpi

[max]% module load maxwell comsyl

[max]% mpirun -np 4 python /software/comsyl/comsyl1env/comsyl/comsyl/calculateAutocorrelation.py /software/comsyl/comsyl1env/comsyl/comsyl/configurations/septest_cm_new_u18_2m_1h.json

Sample batch script

#!/bin/bash
#SBATCH --partition=all
#SBATCH --time=2-00:00:00
#SBATCH --job-name=comsyl
#SBATCH --output=sbatch-comsyl.out
#SBATCH --nodes=1
unset LD_PRELOAD

source /etc/profile.d/modules.sh
module purge
module load maxwell comsyl

mkdir -p calculations # output directory used by samples

# sample 1: runs for a few minutes
time mpirun -np 4 --mca pml ucx python /software/comsyl/comsyl1env/comsyl/comsyl/calculateAutocorrelation.py \
                      /software/comsyl/comsyl1env/comsyl/comsyl/configurations/septest_cm_new_u18_2m_1h.json

# sample 2: runs for several hours. this sample required petsc configuration --with-64-bit-indices
mpirun -N $(( $(nproc) / 2)) --mca pml ucx python /software/comsyl/comsyl1env/comsyl/comsyl/calculateAutocorrelation.py \
                               /software/comsyl/comsyl1env/comsyl/comsyl/configurations/high_flux/hf_new_u18_1m_3h.json

Building comsyl

Installation instructions for comsyl can be found at https://github.com/mark-glass/comsyl/wiki/Install-COMSYL. We had to deviate a bit from the proposed procedure:

#
# base environment
#
instdir=/software/comsyl
scrdir=/scratch/$USER
mkdir -p $instdir $scrdir

cd $instdir
virtualenv-3.6 -p python3 --system-site-packages comsyl1env
source $instdir/comsyl1env/bin/activate
python -m pip install --upgrade pip

# *** changed:
# petsc uses a bit of deprecated code, so downgrade
pip install numpy==1.18.4 

module load mpi/opnempi-x86_64
export PYTHONPATH=/usr/lib64/python3.6/site-packages/openmpi

# 
# srxraylib
#
cd $scrdir
git clone https://github.com/lucarebuffi/srxraylib
cd srxraylib
python setup.py build
pip install -e . --no-binary :all:
cd $scrdir
rm -rf srxraylib

#
# syned
#
cd ..
git clone https://github.com/lucarebuffi/syned
cd syned
python setup.py build
pip install -e . --no-binary :all: # if fails use instead: python setup.py develop 
cd $scrdir
rm -rf syned

#
# comsyl
#
cd $instdir
git clone https://github.com/mark-glass/comsyl
cd comsyl/
python setup.py build
python setup.py develop
# *** changed:
# just copy the python modules into PYTHONPATH
cp -riv comsyl comsyl.egg-info ../lib/python3.6/site-packages/

# *** changed: patch comsyl
# numpy linspace expects an integer for the number of evenly spaced samples
# dim_x * resample_x is however a float which non-ancient numpy versions won't accept
#
diff $instdir/comsyl1env/lib/python3.6/site-packages/comsyl/waveoptics/Wavefront.py 
$instdir/comsyl1env/lib/python3.6/site-packages/comsyl/waveoptics/Wavefront.py~
489c489
< new_x = np.linspace(x_min, x_max, int(dim_x * resample_x))
---
> new_x = np.linspace(x_min, x_max, dim_x * resample_x)
503c503
< new_y = np.linspace(y_min, y_max, int(dim_y * resample_y))
---
> new_y = np.linspace(y_min, y_max, dim_y * resample_y)

#
# *** changed: petsc/slepc
# the combination of petsc, slepc from the installation instruction is not valid
# versions of petsc, slepc should match
# - use a recent version of petsc, slepc
# - add --with-petsc4py=1, also build the python module
# - add --with-64-bit-indices, some of the samples won't run without
# - don't optimize for hardware 
cd $instdir
wget https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.15.2.tar.gz
tar xf $scrdir/petsc-3.15.2.tar.gz 
cd petsc-3.15.2/
export PETSC_DIR=`pwd`
export PETSC_ARCH=arch-linux2-c-opt
./configure --with-mpi-f90=/usr/bin/mpif90.openmpi --with-mpi-dir=/usr/lib64/openmpi 
--with-scalar-type=complex 
--with-debug=0 COPTFLAGS='-O3' CXXOPTFLAGS='-O3' FOPTFLAGS='-O3' --with-petsc4py=1 
--with-64-bit-indices
make -j 8
export PYTHONPATH=$instdir/petsc-3.15.2/arch-linux2-c-opt/lib:$PYTHONPATH

#
# slepc
#
cd $instdir
wget https://slepc.upv.es/download/distrib/slepc-3.15.1.tar.gz
tar xf slepc-3.15.1.tar.gz 
cd slepc-3.15.1/
export SLEPC_DIR=`pwd`
make SLEPC_DIR=$instdir/slepc-3.15.1 PETSC_DIR=$instdir/petsc-3.15.2 PETSC_ARCH=arch-linux2-c-opt

#
# srw ... use install from comsyl
#
- nothing done