1599993 : Running Scripts on the Cluster from a Windows Computer¶

Created: 2026-05-11T10:27:43Z - current status: new¶

Here’s the anonymized summary of the reported issue and a suggested workflow:

Summary of the Issue¶

A user is performing tomographic reconstruction using TomoPy and ASTRA toolboxes on the Maxwell cluster. They currently: 1. Store analysis scripts in a GPFS path (beamtime directory). 2. Submit SLURM array jobs (e.g., sbatch --array=1-249) to process multiple tomograms in parallel. 3. Run reconstructions via Python scripts (e.g., reconstruction_align_auto_cen_ave1.py) with dynamically generated configuration files.

The user wants to centralize script management by moving all analysis scripts to a Windows computer at the beamline, allowing users to modify them locally before submitting jobs to the cluster. They seek recommendations for a streamlined workflow to achieve this.

Suggested Workflow¶

1. Script Storage & Version Control¶

Option A: Shared Network Drive
Store scripts on a Windows-accessible network drive (e.g., GPFS or a dedicated share like \\desy\group\beamline\scripts).
Ensure the cluster can access the same path (e.g., via /gpfs/desy/group/beamline/scripts).
Pros: Simple, no sync required.
Cons: Requires network access; may need IT support for cross-platform permissions.
Option B: Git Repository
Host scripts in a Git repository (e.g., GitLab/GitHub) and clone them to both the Windows machine and cluster.
Users modify scripts locally, commit changes, and pull updates on the cluster before submission.
Pros: Version control, traceability, and offline work.
Cons: Requires Git knowledge; sync step needed.

2. Job Submission from Windows¶

Option A: Remote Submission via SSH
Use PuTTY/WSL (Windows) to SSH into the Maxwell login node and submit jobs directly: bash ssh [USERNAME]@max-display.desy.de "cd /path/to/scripts && sbatch job_script.sh"
Pros: No additional tools needed.
Cons: Manual process; error-prone for non-experts.
Option B: Web-Based Submission (Recommended)
Use Maxwell’s web portal (if available) or a custom web interface (e.g., Flask/Django) to:
1. Upload/modify scripts via a browser.
2. Automatically sync scripts to the cluster.
3. Submit jobs via a form (e.g., specifying array ranges, parameters).
Pros: User-friendly; reduces errors.
Cons: Requires development effort.
Option C: Script Automation
Write a wrapper script (Python/Bash) on the Windows machine to:
1. Copy scripts to the cluster (e.g., via scp or rsync).
2. Submit jobs via SSH. Example: ```bash
Windows (PowerShell)¶

scp -r scripts/ [USERNAME]@max-display.desy.de:/gpfs/beamtime/scripts/ ssh [USERNAME]@max-display.desy.de "cd /gpfs/beamtime/scripts/ && sbatch job_script.sh" ```

3. Dynamic Configuration Handling¶

Template-Based Configs
Store a template config file (e.g., rec_parameters_minimal.config) in the shared location.
Use sed (as in the current script) or Python’s string.Template to replace placeholders (e.g., xxx → scan number) at runtime.

4. Error Handling & Logging¶

Centralized Logs
Direct SLURM output/error logs to a shared directory (e.g., /gpfs/beamtime/logs/).
Use tail -f or a web dashboard to monitor logs in real time.

5. Example Workflow¶

User edits scripts on Windows → saves to shared drive/Git repo.
Script syncs changes to the cluster (automated or manual).
User submits job via SSH/web interface.
Cluster processes jobs and writes logs to a shared location.

Key Recommendations¶

Start Simple: Use Option A (shared drive + SSH submission) for quick implementation.
Add Git: Migrate to Option B (Git) for better version control.
Automate: Develop a wrapper script (Option C) to reduce manual steps.
Leverage Maxwell Resources:
Attend the HPC Seminar on Using Maxwell for best practices.
Check the Maxwell documentation on workflows for advanced automation (e.g., Swift).