Skip to content

Memory Usage

About Memory

All jobs need to use a certain amount of memory or RAM in order to run. We use the terms memory and RAM interchangeably. Neither term refers to the space occupied by files on hard drives.

The primary command to use to get information about your SLURM jobs is the sacct command. One of the things it displays is a job state. If the STATE field from the sacct command is OUT_OF_MEMORY it means that your job has run out of RAM, and you will need to resubmit your job with a larger RAM request.

Requesting Additional Memory

When you submit a job with sbatch, or run srun to get an interactive job on a compute node, you are by default allotted 5GB of RAM for your job. (And one CPU core.) This is true on both of our clusters: JADE & JHPCE.

If you are running a larger job and need more than the 5GB of RAM, you can request more (or less) RAM via two command options:

    --mem : memory per node (for all cores used)
    --mem-per-cpu : memory per core (harder to accurately estimate)

Some examples of using these options:

# This would give your batch job a total of 10GB of RAM
sbatch --mem=10G job1.sh

# This would give your interactive session a total of 20GB of RAM & 4 cores
srun --mem-per-cpu=5G --cpus-per-task=4 --pty --x11 bash

Tools to use to get RAM usage information:

sacct & sstat

The sacct command is used to query the SLURM job accounting database, usually for jobs which have ended (one way or the other). Of course, if a job has not started or finished, sacct cannot show you things it does not yet know. The sstat program can look at stats about running jobs.

This page has instructions & examples to help you use sacct & sstat:
https://jhpce.jhu.edu/slurm/tips-sacct

seff & reportseff

seff is a program which looks up accounting data using the sacct command to show some stats for a single completed job. reportseff allows you to look over the information for a number of jobs.

This page has instructions & examples:
https://jhpce.jhu.edu/slurm/tips-reportseff

Estimating RAM usage

There is, sadly, no easy formula to know ahead of time how much RAM a job will need when working with large data. Every program has its own way of consuming RAM, and the size or complexity of a dataset also matters. An iterative process is required to determine the best value to use for your jobs.

Here are some tips to help find good values for your job:

  • You can run a test job on a small subset of data, then a larger one. From this kind of data you should be able to extrapolate the amount of RAM your full job will need. The seff and reportseff commands are useful here.
  • One good place to start looking for a minimum is to look at the size of the files you will be reading in. Add a bit extra, as a starting point. If your job is reading in a 20GB image file, you may want to ask for 25GB or RAM.
  • However the way your job processes data and impacts the RAM used can vary wildly. The code you write in a batch script, and all of the code in libraries or applications spawned by your script, can use fractions or multiples of any data file's size. Your code may wind up reading in the file one chunk at a time, then releasing that memory as it finishes, then read in another chunk, and will not grow as much as if it read it all in at once. Orrrrrr your code might copy the original data into a second array, thereby doubling the amount of space needed. It all depends.
  • You can run sacct to gather info on a completed job, where JOBID is its number.

    sacct -j JOBID -o JobID,JobName,ReqTRES%40,MaxVMSize,MAXRSS,State%20

  • You can run sstat to gather info on a running job.

    sstat -j JOBID -a -o JobID,MaxVMSizeNode,MaxVMSize,AveVMSize,MaxRSS,AveRSS,MaxDiskRead,MaxDiskWrite,AveCPUFreq,TRESUsageInMax

Impacts of RAM requests

Try to make your RAM request slightly higher than your expected usage.

  • Too low and your job will get killed for exceeding your request
  • Too high and
    • your job may take longer to get scheduled,
    • you’ll be squatting on RAM that others can use,
    • you'll be able to run fewer jobs if your partition has a quota, and
    • the job will cost more.

To get a feeling for what amounts of RAM are available on compute nodes, run the slurmpic command. You will see the total core and RAM on each node, as well as current core and RAM usage/availability. If you want to request 750GB, you will see how many/few nodes can even be considered candidates for your job.

About the three RAM-related columns slurmpic prints:

  • TOT_MEM - Total amount SLURM can use
  • FREEMEM - Amount remaining that SLURM jobs can use. (When you request RAM, it is reserved for you. SLURM takes TOT_MEM and adds or subtracts the per-job requested RAM values to arrive at FREEMEM.)
  • SYSFREEMEM - You can ignore this. It is the value that the node says is unused by any process. UNIX is aggressive about using spare memory to cache and buffer data. Such use is only cleared when the memory is actually needed by a different program. (It can reveal, though, if users requested lots of RAM but then didn't use it (yet).)