Factors Affecting Job Scheduling¶
TL;DR¶
Request the smallest and/or fewest resources which will allow your job to run to completion.
Choose the appropriate partition for your work. Partitions have different rules, which are explained here.
"Smaller" jobs "fit" into more slots between other jobs than "larger" jobs, so request only the resources you need. You might also consider whether you can divide up your work. You might also want to direct your job to the most appropriate partition. For example, the JHPCE cluster has an "interactive" partition for small jobs.
Understand that we have implemented various policies which impact the ordering of jobs which are otherwise equivalent in terms of resources requested.
If you have used a lot of resources in public partitions recently, (in the last week, especially over the last three days), your jobs will have a lower priority than those of someone who has consumed less.
If the scheduler has been able to determine an estimated start date for your job, it will be shown in the output of
squeue --me --start
Overview - The Problem¶
A job consists of at least three resources:
- maximum job duration (time),
- maximum amount of memory (RAM), and
- maximum number of CPU cores (CPU).
Other resources might include a partition, a device like a GPU card, or a reservation.
You can think of the total of these resource requests as a multiply-dimensioned box of a certain "size" defined by adding together the components. Then visualize looking at a chart where you have to fit each box into your collection of compute or GPU nodes, at a specific time, for a certain duration.
One of SLURM's primary functions is to schedule jobs, which consists of two tasks:
- Technical: Finding the right "sized" possible slots for each job either now or into the future. The "smaller" the slot, the easier it will be to find.
- Policy: Choosing specific jobs to place into specific slots according to other factors chosen by cluster administrators.
Over the decades, organizations using tools like SLURM have desired both
- the highest efficiency utilization of available hardware, and
- the ability to set policies about who gets which resources, and in what order.
Our Choices¶
SLURM has many "knobs" that can be tuned to try to reach those goals. Some of the most important ones are described below. There are a number of vendor documents which document scheduling. See the "Workload Prioritization" and "Slurm Scheduling" sections at this site.
Priority access for Primary Investigator group members¶
Our clusters (currently JHPCE and JADE) contain compute node hardware purchased, in the main, by primary investigators. Most of them choose to share their nodes with the larger community because they value collaboration and doing so reduces the costs of operating their nodes. We group nodes together into SLURM partions, each of which can have different rules applied to them. Those rules are part of implementing policies, such as giving members of a PI's research teams priority access to the hardware that team paid for. As a general rule, this has resulted in having two sets of partitions: "private" ones for PI teams and "public" ones for everyone to use. (Details about partitions can be found at here
Resource Limits¶
Public partitions have rules in place which restrict the amount of each of the main types of job resources. For example, JHPCE's "shared" partition's definition includes a maximum job duration as well as a Quality of Service (QOS are described here. The QOS for the shared partition limits the amount of RAM and CPUs that a user can consume at any one time.
Per-job Priorities - Multifactor Job Priority¶
We have enabled a common SLURM plugin called "Multifactor Job Priority". Multiple factors are used to assign a single priority value to each job. This is described in the Multifactor Job Priority document.
(This priority is only used to decide which jobs to dispatch first. It is not used to set a UNIX process nice value on the processes created by jobs out on the compute nodes.)
Once a job starts running, its priority no longer has much meaning.
The job's priority is an integer that ranges between 0 and 4,294,967,295. The larger the number, the higher the job will be positioned in the queue, and the sooner the job will be scheduled and started. A job's priority, and hence its order in the queue, can vary over time.
The final priority is determined by multiplying pairs of (weights and factors) and adding the results. Factors range from 0.0 to 1.0. Weights range from 0 to 65,533.
Currently we are using three components: Age, Fairshare and Partition
Tip
You can see pending job's priority values and the contributors to the final value with the sprio command. This sorts jobs by total prio, partition, user.
sprio -S -y,p,u | less
Tip
A better formatted of that command which prints only the factors we are currently using1 is:
sprio -o "%.15i %9r %.8u %.10Y %.10A %.10F %.10P" -S -y,p,u
Tip
You can change the priority of your jobs among your jobs with scontrol commands like top and nice. See this document for details.
Fairshare¶
To help provide equitable access to the public partitions of the cluster, the FAIRSHARE priority component is based on your recent usage of those partitions. If you have used fewer CPU minutes than someone else in the last week, then your jobs will receive a higher fairshare value.
The fairshare priority is the result of multiplying a weight stored in a variable, PriorityWeightFairshare, and a factor which is derived from the accounting database.
Tip
You inspect fairshare values for ALL users with this command:
sshare -a | sort -k7nr | less
You should focus on the values in the right-hand-most column. Heaviest users of the cluster in recent days have values closer to 0.0. People who haven't run any jobs lately will have values closer to 1.0. Jobs submitted by the latter will be given higher fairshare priority values.
Age¶
In addition to fairshare, any pending job will accrue AGE priority over time. Currently (20240217) this maxes out to 100 over the course of a week.
Job arrays which started running tasks many days ago will wind up with high age priority values for all of their future tasks. You can see that fairshare somewhat counteracts that age advantage.
If you decide that you want to change something about a pending job, consider whether you can do so using scontrol commands as described here instead of killing the job with scancel and resubmitting it. That would preserve the age priority your job has accrued.
Partition Priority¶
We have set this experimentally on the interactive JHPCE partition to try to aid in quick access to (small) interactive sessions.
Backfill¶
In addition to the main scheduling cycle, where jobs are run in the order of priority and availability of resources, all jobs are also considered for "backfill". Backfill is a mechanism which will let jobs with lower priority score start before high priority jobs if they can fit in around them. For example, if a higher priority job needs 30 cores and it will have to wait 20 hours for those resources to be available, if a lower priority job only needs a couple cores for an hour, Slurm will run that shorter job in the meantime. This GREATLY enhances utilization of the cluster.
For this reason, it is important to request accurate walltime limits for your jobs. If your job only requires 2 hours to run, but you request 24 hours, the likelihood that your job will be backfilled is greatly lowered.
-
This command's output will be incomplete if we begin using other priority factors. ↩