1609928 : Help with parallelization for analysis script needed

Created: 2026-06-17T14:47:40Z - current status: new

Anonymized Summary & Solution

Issue

A researcher wants to efficiently run a Python-based ligand-detection tool (FoxSight) on a Slurm-managed HPC cluster (Maxwell), comparing its performance against other tools. The current implementation uses an SGE array job, but they need guidance on optimizing it for Slurm, particularly for parallel execution across multiple dataset comparisons.

Key points: - The task involves processing many independent .ccp4 map files in parallel. - Their current approach uses ProcessPoolExecutor (Python multiprocessing) within a single-node array job. - They seek advice on whether Slurm supports concurrent multi-threaded tasks per node (like SGE does).

Summary

The user needs help adapting their existing array job + Python multiprocessing workflow from SGE to Slurm for efficient parallel processing on Maxwell.

Solution

  1. Slurm Array Jobs:
  2. Use #SBATCH --array=<start>-<end> to distribute tasks across nodes (already correctly implemented).

  3. Each array task runs one subprocesswrapper.py instance (good for isolation).

  4. Parallelism Within a Node:

  5. Unlike SGE, Slurm does allow multi-threading within a single array task.
  6. Modify the script to use --ntasks-per-node and --cpus-per-task to enable intra-node parallelism: bash #SBATCH --cpus-per-task=4 # Allocate 4 CPUs per task

  7. Adjust nof-processes in subprocesswrapper.py to match --cpus-per-task.

  8. Optimizations:

  9. Replace ProcessPoolExecutor with ThreadPoolExecutor if tasks are I/O-bound (but ensure thread safety).

  10. Avoid oversubscribing CPU cores (match --cpus-per-task to actual usage).

  11. Example Slurm Script: ```bash #!/bin/bash #SBATCH --partition=allcpu #SBATCH --time=12:00:00 #SBATCH --nodes=1 #SBATCH --cpus-per-task=4 # Enable 4-way parallelism per task #SBATCH --array=351-781 #SBATCH --output=slurm-%j_%a.out

unset LD_PRELOAD source /etc/profile.d/modules.sh module purge

echo "Task ${SLURM_ARRAY_TASK_ID} starts on $(hostname)" python3 subprocesswrapper.py \ --force \ --target-dir '[REDACTED]' \ --target-pattern './//*.ccp4' \ --job-id ${SLURM_ARRAY_TASK_ID} \ --nof-processes $SLURM_CPUS_PER_TASK \ # Dynamically set workers --log-file '[REDACTED]/results.log' ```

  1. Notes:
  2. Ensure FoxSight binaries are accessible via $PATH/$LD_LIBRARY_PATH.
  3. Monitor resource usage with sacct or seff to validate efficiency.

This approach balances inter-node distribution (via arrays) and intra-node parallelism (via --cpus-per-task), aligning with Slurm’s design.