DUST Migration for IDAF¶
Since 2025-03-05, all groups have been migrated to the new directory structure on DUST!
In order to establish a unified namespace of DUST in the IDAF, there will be a breaking change. While we foresee benefits for users in the future, there is of course the inconvenience of the migration.
Executive summary:
- The new mountpoint is:
/data/dust - Single location for user directories:
/data/dust/user/ACCOUNT - Group directories will keep the group name:
/data/dust/group/GROUP/SPACE_NAME- Exception: The directories for IAXO, ALPS2 & MADMAX will be consolidated in
/data/dust/group/axion/
- Exception: The directories for IAXO, ALPS2 & MADMAX will be consolidated in
- Users with multiple user directories will be merged to a single directory
- Migration will be carried out per user group, access to DUST is not possible during the migration for this group
- You do NOT have to perform any data migration, this will be handled by IT
Migration Status¶
| Date | Group | Path | Status |
|---|---|---|---|
| 2024-12-02 | Maxwell | /gpfs/dust/maxwell |  |
| 2024-12-02 | AF-M | /nfs/dust/m | |
| 2024-12-03 | IT | /nfs/dust/it | |
| 2024-12-13 | H1, HERMES, ZEUS, FHLABS | /nfs/dust/h1, /nfs/dust/hermes, /nfs/dust/zeus, /nfs/dust/fhlabs | |
| 2024-12-17 | BELLE | /nfs/dust/belle1, /nfs/dust/belle2 | |
| 2025-01-20 | CMS | /nfs/dust/cms | |
| 2025-03-05 | ATLAS, ILC, LUXE, IAXO, ALPS2, MADMAX | /nfs/dust/atlas, /nfs/dust/ilc, /nfs/dust/luxe, /nfs/dust/iaxo, /nfs/dust/alps2, /nfs/dust/madmax | |
Time Schedule¶
| Date | Action |
|---|---|
| 2024-12-02, 8:00 AM to 3:00 PM | Migration of /gpfs/dust/maxwell to new structure |
| 2024-12-03, 8:00 AM to 10:00 AM | Migration of /nfs/dust/it to new structure |
| 2024-12-13, starting 8:00 AM | Migration of /nfs/dust/h1, /nfs/dust/hermes, /nfs/dust/zeus & /nfs/dust/fhlabs to new structure |
| 2024-12-17, 8:00 AM to 1:42 PM | Migration of /nfs/dust/belle1 & /nfs/dust/belle2 to new structure |
| 2025-01-20, 8:00 AM to 6:00 PM | Migration of /nfs/dust/cms to new structure |
| 2025-03-05, 8:00 AM to 8:00 PM | Migration of /nfs/dust/atlas, /nfs/dust/ilc, /nfs/dust/luxe, /nfs/dust/iaxo, /nfs/dust/alps2, /nfs/dust/madmax to new structure |
Procedure¶
To create the new directory structure, all data on DUST has to be copied to the new location.
IT will take care of the data migration, you do not have to copy data!
The data migration will be performed per user-group, e.g. all of /nfs/dust/cms or /gpfs/dust/maxwell.
During the migration period, no access to DUST will be possible. The migration differs for users with single- and multiple user directories.
Users with a single user directory¶
If you have a single user directory, e.g. /nfs/dust/GROUP/user/ACCOUNT, your data will be copied 1:1 to /data/dust/user/ACCOUNT.
Your quota will be managed by the same administrators as before.
This is the vast majority for all users on DUST.
Users with multiple user directories¶
If you have multiple user directories, e.g. /nfs/dust/GROUP1/user/ACCOUNT and /nfs/dust/GROUP2/user/ACCOUNT, your data will be merged into a single directory /data/dust/user/ACCOUNT.
To avoid conflicts, your data will be stored in subfolders with the name of the group and a .merged suffix:
/nfs/dust/GROUP1/user/ACCOUNT -> /data/dust/user/ACCOUNT/GROUP1.merged
/nfs/dust/GROUP2/user/ACCOUNT -> /data/dust/user/ACCOUNT/GROUP2.merged
User with directories in CMS, MAXWELL and ILC
You have the directories /nfs/dust/cms/user/ACCOUNT, /gpfs/dust/maxwell/user/ACCOUNT & /nfs/dust/ilc/user/ACCOUNT
Your new user directory location will be: /data/dust/user/ACCOUNT
The data from your user directories are stored in subfolders: /data/dust/user/ACCOUNT/af-cms.merged, /data/dust/user/ACCOUNT/maxwell.merged & /data/dust/user/ACCOUNT/af-ilc.merged
Due to quota management purposes, you can only belong to a single group. We'll determine this group based in your primary group of the DESY account. You'll also receive an email with this information and can request to change this before the migration.
Your new quota for /data/dust/user/ACCOUNT will be the sum of the previous user directories.
Depending on the migration schedule, your data will be migrated earlier than your actual group. In this case, we will provide compatability symbolic links. Symbolic links will point from the previous locations to the new location, see below for an example.
User with directories in CMS & MAXWELL, MAXWELL will be migrated first
You have the directories /nfs/dust/cms/user/ACCOUNT & /gpfs/dust/maxwell/user/ACCOUNT
As Maxwell will be migrated first, your CMS directory must be copied as well.
Your new location will be /data/dust/user/ACCOUNT.
To allow easy access for your colleagues from CMS until CMS itself will be migrated, /nfs/dust/cms/user/ACCOUNT will be replaced with a symlink pointing to /data/dust/user/ACCOUNT/af-cms.merged
Users with multiple user directories are a minority, this will only affect <= 80 users on DUST.