1620826 : /pnfs/desy.de/belle/local not mounted/populated on BIRD execute nodes

Created: 2026-07-28T14:55:19Z - current status: new

Summary

A user reports inconsistent access to /pnfs/desy.de/belle/ paths between the interactive node (naf-belle21) and HTCondor batch nodes (e.g., batch1453.desy.de). While the full directory structure (including /belle/local/... subdirectories) is visible on naf-belle21, batch jobs fail with "file does not exist" errors because /pnfs/desy.de/belle/ appears empty on the execute node. Some files work, while others fail due to missing mounts.

Possible Cause

The issue likely stems from incomplete or inconsistent PNFS namespace exports/mounts across compute nodes during data migration or configuration updates. Batch nodes may lack proper NFS/PNFS client setups for certain subtrees.

Next Steps

  1. Verify Mount Consistency: Check if /pnfs/desy.de/belle/local is exported correctly on storage servers and mounted uniformly across all batch nodes.
  2. Test Access Patterns: Identify why some files resolve successfully—compare working vs. failing paths for differences in permissions or mount points.
  3. Engage Storage Admins: Confirm whether recent data migrations affected subtree visibility and request validation of PNFS configurations.

Note: All identifiers (names, usernames, hostnames) were replaced with generic descriptions. No original metadata remains.