Storage-Infrastructure¶
there a number of storage systems attached to the cluster. As a rule of thumb:
- Store everything of crucial importance in your home-dir or group- or experiment-specific GPFS folder when available.
- Archive data in dCache if available for your group.
- Store everything easily reproducible and in particular when of temporary nature in DUST
More details can be found below
IMPORTANT
Any files without OWNER (account does not exist anymore or is expired) will be removed from DESY BEEGFS, DUST, HOME, GPFS-FS-SCRATCH without prior notifications! Deleted files are irrevocably lost. Similar policies apply for group-storages, please verify with your admins if in doubt.
Overview of storage systems¶
The administration and troubleshooting slightly varies across installations. The table below summarizes responsibilities. If in doubt it's always a good idea to contact uco@desy.de.
| Filesystem | Remarks | Mounts | Contact |
|---|---|---|---|
| AFS | Good for static data/documents and software. Not good for multithreaded applications and especially batch jobs. | /afs/desy.de/ | DESY IT (uco@desy.de) for questions about increasing quote, general problems |
| dCache | Good for mass storage of scientific data. Not suitable for volatile data. | /pnfs/desy.de/ | DESY IT (maxwell.service@desy.de) to arrange (missing) mounts. uco@desy.de for general questions |
DEPRECATED | use DUST instead | ||
| sync&share | Good for data sharing. | none | DESY IT uco@desy.de for all issues |
| DUST | Good for software & temporary/reproducible data | /data/dust | DESY IT (maxwell.service@desy.de) |
| CFEL GPFS | Fast. Good for scientific data of CFEL. | see below | CFEL IT / DESY IT. DESY CFEL admins for access/usage issues. |
| CSSB GPFS | Fast. Good for scientific data from CSSB. Lifetime of data is up to CSSB policy. Get in touch with CSSB admins for details! | /gpfs/cssb | CSSB IT / DESY IT. it@cssb-hamburg.de for access/usage rights & quota management. maxwell.service@desy.de for issues. |
| EXFEL GPFS | Fast. Good for scientific data from EuXFEL experiments and analysis. | see below | EXFEL IT / DESY IT. EXFEL admins for access/usage issues. maxwell.service@desy.de for technical issues |
| FS GPFS | Fast. Good for scientific data from PETRA III and FLASH experiments and analysis. | see below | ASAP3 Team, it-asap3@desy.de for all issues. |
| Scratch / TMP | Scratch. Limited space and subject to erasure without prior notice. | /scratch /tmp | DESY IT Unmanaged temporary space. |
| HOME | Fast. With snapshots. Best space for everything important and reasonably small. | /home | DESY IT maxwell.service@desy.de for all issues. Quota will not be extended! |
Characteristics of storage systems substantially differ for different storage systems. Tables below give a rough overview. Contact maxwell.service@desy.de for open questions.
| Filesystem | Mount Point | Quota | Size | Lifetime | tokens | Backup | Snapshot | Network | Remote |
|---|---|---|---|---|---|---|---|---|---|
| AFS | /afs/ | per Volume | per Volume | unlimited | yes | yes1 | yes2 | eth | yes |
| DUST | /data/dust/ | yes | ~2PB | unlimted | no | no | no | ib | no |
| dCache | /pnfs | no | unlimited | unlimited | no | no | no | eth | yes |
| sync&share | none | n/a | unlimited | unlimited | no | no5 | no | eth | yes |
| GPFS CFEL | /gpfs/cfel | no | 1.6PB | no | no | yes | ib | no | |
| GPFS CSSB | /gpfs/cssb | 50 TB | 11PB | unlimited | no | no | yes3 | ib | no |
| GPFS EXFL | /gpfs/exfel/d | 47PB | |||||||
| /gpfs/exfel/data | 1.2PB | ||||||||
| /gpfs/exfel/exp | 32GB | ||||||||
| /gpfs/exfel/u | 3.4PB | ||||||||
| GPFS FS | /asap3 | no | 15PB | long | no | yes | yes4 | ib | no |
| /gpfs/petra3/scratch | no | 40TB | 3 month | no | no | no | |||
| /gpfs/petra4/scratch | no | 120TB | |||||||
| scratch | /scratch | no | few GB | none | no | no | no | - | no |
| HOME | /home on GPFS | 30GB hard | 70TB | account | no | no | yes5 | ib | no |
footnotes
- For information about backup & recovery: check IT-Services
- AFS-snapshots are located in <afs-home>/.OldFiles
- GPFS CSSB Snapshots are located in /gpfs/cssb/.snapshots/@<time-stamp>
- GPFS-ASAP3 Snapshots are located in /asap3/.snapshots/@<time-stamp>
- GPFS-Home snapshots are located in /home/.snapshots/@<time-stamp>/$USER
Where to store Scientific data¶
Available for everyone are AFS, DUST, Desycloud and dCache.
- AFS is secure, but suitable only for rather small data volumes.
- Desycloud offers significantly more space, but uploads or downloads are not very fast.
- DUST and GPFS-Scratch are fast, but exclusively for temporary or reproducible data and software installations without any level of security. Please note the policy!
- dCache: is the only option for long-term storage of larger amounts of data. If your group doesn't have dCache storage space but would like to "buy in", get in touch with osm.service@desy.de .
- GPFS home-directories: 30GB hard limit (can't be increased), don't store data in the home-directory.
In addition to generic storage resources, some group specific resources are available:
- GPFS-CSSB: space dedicated to CSSB
- GPFS-FS resources: space dedicated to FS-experiments.
- GPFS-CFEL resources: space dedicated to CFEL.
- GPFS-EXFL resources: space dedicated to European XFEL.
- IF your group needs large amounts of fast, secure storage space, a group-owned GPFS appliance might be a solution. Get in touch with maxwell.service@desy.de if you need to know more.
Where to store Software¶
Most applications are small enough to be deployed in (almost) arbitrary locations. Suitable storage systems are
- AFS: globally accessible space. Good for software needed on different platforms. Keep in mind, that restrictive ACLs will cause problems on Maxwell! AFS is not suitable for multi-host application, it will have horrible side-effects on performance!
- GPFS-home: apart from the space limitations, well suited for software installations. Don't use it for group-shared installations!
- DUST: well suited for software installations. For applications shared within a group use group directories. Contact maxwell.service@desy.de if you need a new group directory.
- dCache DO NOT use for software installations.
Where to store Documents¶
Assuming that documents are usually small
- AFS: secure with flexible ACLs. Globally accessible. Prime choice for documents.
- Desycloud: well suited for sharing documents. Globally accessible. Not convenient to use in the HPC environment.
- GPFS-home: secure & fast, but limited in space and not easily accessible outside maxwell
- Don't store documents on DUST, dCache or GPFS (except for documenting experiments).
Where to store Temporary data¶
- DUST: designed as scratch, it's perfect for anything reproducible or temporary data.
- scratch: local space, but fairly limited volumes.
- Don't store temporary data on dCache or AFS.