A shared scratch area is available in the cluster at /scratch/<youruserid>. It is shared across all the nodes and hence the data written to this folder (sub-folders included) will be seen (almost) simultaneously from all the other nodes. This space should be used to store TEMPORARY job data ONLY. Please do not store any data that is critical in the scratch space as this is not under any backup / restore schedule. Also, please note that your job needs to handle file writes to this area smartly so jobs do not overwrite each other's input/output/data files.
Space usage of the /scratch volume is metered and cleaned to prevent runaway workloads from impacting other users in the cluster. The following rules are in place:
- A user's scratch (/scratch/<userid>) directory has a baseline limit of 500GB
- A user can burst over this 500GB limit for up to 14 days to a maximum of 5TB
- If a user remains over 500GB past 14 days then additional data writes are denied
- A user can reset their burst usage clock by reducing usage under 500GB for 10 minutes.
- Data older than 60 days are cleaned every day. Be sure to move your data off this space as soon as your job completes
If you need more permanent storage beyond what your home directory and /scratch provide, check with your PI/manager to see if your group already has an existing share you can use. If not, you can request a new research network data share which will be mounted on the cluster under /data.”
Research Network Drive - Information Services (service-now.com)
Details on responsible use policy at CCHMC can be found here:
To see your Scratch Usage at any time you can issue the "scratchusage" command on the login nodes.
wei2dt@bmiclusterp2:~$ scratchusage Details on the policies and workings of scratch volume can be found at: https://hpc.research.cchmc.org/node/14 Scratch Usage: 0GB Base Limit: 500GB Burst Limit: 5120GB Burst Duration: 14 Days 500GB Base quota is not currently in violation. wei2dt@bmiclusterp2:~$ ./scratchusage Details on the policies and workings of scratch volume can be found at: https://hpc.research.cchmc.org/node/14 Scratch Usage: 785GB Base Limit: 500GB Burst Limit: 5120GB Burst Duration: 14 Days Base quota of 500GB has been exceeded: -Base quota was last exceeded on: Fri Jun 14 15:39:39 EDT 2024 -Scratch locked for further writes on: Fri Jun 28 15:39:39 EDT 2024 **FYI: You must reduce usage on your scratch volume below 500GB for 10 min to reset your usage clock.