You can use the following file systems:
|Size||>300 TByte (300,000 GByte)||>900 TByte (900,000 GByte)||>150 TByte (150,000 GByte)||>100 GByte per node|
|Access time||normal||fast||fast||very fast, low latency|
|Accessible files||global from all nodes||global for all nodes||global for all nodes||local disks of the compute node, file is not accessible from other nodes|
|Available files||permanent||after 8 weeks files will be deleted without replacement and without announcement||for the project period||only while the job is running|
|Quota||15 GByte, after request maybe more||10 TByte or 2Mio. files, after request maybe more||after request||none|
Snapshots (see below) +|
daily tape backup (desaster recovery)
The home directory should be used for all files that are important and need to be stored permanently. The
/home filesystem is based on a rather fault-tolerant disk configuration, which comes at the price of being slower and every user can only store a small amount of data here. Standard quota is currently 15 GByte. In well reasoned cases and on request, this quota can be increased. The folder /home/$USER (“Home”) is created with each user account. It is accessible by the environmental variable
Our filesystem automatically creates periodic snapshots of the home folders, allowing you to access (and restore) older versions of your files without assistance by the admins. Snapshots are saved to the hidden folder
.snapshots (you would not see this folder listed even by an 'ls -la'). Nonetheless, you can go to that hidden folder by explicitely “
cd .snapshots” (<TAB>-completion does not work either, you have to fully type
.snapshots). Being in
.snapshots/, you can do 'ls -l' and 'cd' as usual, and access former versions (or states) of all your
$HOME data (within the
Files from the snapshot folder still occupy storage space and thus, do affect your quota! Therefore, it is possible your home folder's quota is exceeded, even though the 'df' command still shows less usage. Snapshots cannot be deleted (deleting data creates copies of the snapshot).
Frequent saving and deleting files fills up the snapshot area and requires space at the home folder. If possible, this should thus be avoided.
In urgent cases, the snapshot folder can be deleted by the administrators.
In addition to the snapshots, we do periodic tape backups (currently weekly) of the home folder, but this is for disaster recovery only. Recovery of individual user files is not possible from these tape backups.
Here, almost unlimited disk space is available for all users, but only for a limited time: After 8 weeks the files will be deleted without warning. These files are not backed up by any means.
The disk configuration for the
/work area is geared towards maximum performance and capacity. Of the cluster-wide filesystems,
/work/scratch is the fastest in terms of throughput.
Standard quota is currently 10 TByte or 2 million files. In well reasoned cases and on request, this quota can be increased.
/work/scratch/$USER (“scratch”) is created with each user account. It is accessible by the environmental variable
The local disks at the individual compute nodes are mounted at “
/work/local” and are to be used during an individual job's calculation. Due to the low latency of the node-local disks, intermediary files can be stored there quite efficiently.
When a job is assigned a certain node and is started there, the folders
/work/local/$SLURM_JOBID/tmp are created on that assigned node. For convenience, two corresponding environment variables
$TMP are set, which you can use in your jobscripts instead of the longer “
At the end of the job, these subdirectories and their content will be deleted automatically. Thus it is imperative to save any final results from
$HPC_LOCAL/ before the end of the job to the
scratch file system! Nor can you use
$HPC_LOCAL for checkpoint/restart files, where later jobs are to continue earlier job's calculations based on these CPR files.
The local disk space of the login nodes has been assigned to the
/tmp directory, as here the Slurm JobID based scheme would not work.
All shared, cluster-wide filesystems above are based on IBM's Spectrum Scale (formerly General Parallel File System). This commercial product can share large disk arrays among thousands of nodes per Infiniband.
Of course, arbitrating read/write requests from such numbers of nodes to individual files will take some more time than accessing local disks. That's why you sometimes see (hopefully short) “hiccups” when doing a
ls -l or the like.
The local disks inside the nodes are usually SATA drives with
ext4, and since jobs mostly have their node(s) exclusively, are faster than the global GPFS.