# Resource Allocation

**Resource allocation** in HPC is the set of resources you are given to use for your jobs. A **resource** in this context is defined as a given number of CPU cores or GPUs, a certain amount of CPU or GPU usage per second, or a given amount of RAM.

There are several approaches to ensuring fair resource management, which include:

- **F**irst **I**n, **F**irst **O**ut (FIFO)
- **Backfill** (smallest job first)
- **Fairshare**, which dynamically adjusts priority based on users' recent usage.

## Slurm

Resource allocation on Monsoon is handled by a system called [slurm](<https://slurm.schedmd.com/overview.html>). Slurm grants access to resources over a given block of time, called an allocation. When you submit a job to Monsoon, you are actually requesting a given set of resources (e.g 1 CPU core, 1GB of RAM, and 1 GPU) over a given time span (e.g 1 hour). Slurm then schedules these blocks through a priority system, described below.

If a job is canceled or finishes early, a gap is created in the schedule between that job and the one that follows it. Depending on the job, this means Slurm can sometimes place a job into a much sooner time slot than it was originally scheduled for. This process is known as **backfilling**.

## Fairshare

**Fairshare** is a value between 0 and 1 that is calculated based on a user's recent resource usage. The *higher* the fairshare value is, the *higher* the priority for scheduling jobs. The *more* resources a user has used recently, the *lower* their fairshare value will be. This means that users who have used fewer resources recently will have a higher priority for their jobs to be scheduled.

More information on the fairshare algorithm can be found in the [slurm documentation](<https://slurm.schedmd.com/fair_tree.html#algorithm>).

## LevelFS

**LevelFS** is a value that represents the level of fairness in resource allocation. The ranges of LevelFS levels and their meanings are as follows:

- LevelFS \< 1: High priority for scheduling jobs
- LevelFS 1 – 10: Moderate priority for scheduling jobs
- LevelFS \> 10: Low priority for scheduling jobs

## TRESRunMins

Separately from fairshare, Monsoon also implements a system for limiting jobs called \\(TRESRunMins\\), which determines the upper bound on how many CPU-minutes a given Slurm account (e.g faculty member or classroom) can use.

This is a variable in Slurm that represents a number assigned to an account (or \\(QoS\\)) which limits the total number of remaining CPU minutes which your running jobs can occupy. Having this feature enabled on Monsoon helps with:

- Flexible resource limiting
- Staggering jobs
- Increasing cluster utilization
- More accurate resource requests

The current value for the limit is 3,000,000. This value is sometimes increased as cluster utilization drops, which allows folks to use the idle cores. To calculate the \\(TRESRunMins\\) for your jobs, multiply the number of CPUs being used by the time limit remaining, then multiply that number by the total number of jobs you are running, like so:

\\(TRESRunMins = \\\#\\ of\\ jobs \* cpus \* time\\ remaining\\)

Examples:

\\(500000 = 24 \* 1\\ \\text{cpu}, 2\\ \\text{week jobs}\\)

\\(500000 = 49 \* 1\\ \\text{cpu}, 1\\ \\text{week jobs}\\)

\\(500000 = 347 \* 1\\ \\text{cpu}, 1\\ \\text{day jobs}\\)

\\(500000 = 21 \* 16\\ \\text{cpu}, 1\\ \\text{day jobs}\\)

\\(500000 = 130 \* 16\\ \\text{cpu}, 4\\ \\text{hr jobs}\\)

\\(500000 = 520 \* 4\\ \\text{cpu}, 4\\ \\text{hr jobs}\\)

To see the current \\(TRESRunMins\\) for a single account or all accounts, use

```bash
sshare -l -A <account name> # single account  
sshare -l # all accounts
```

In the output, the pertinent column will be labeled `CPURunMins` and will be the farthest to the right. This number changes dynamically as jobs change state.

## Checking your Fairshare and LevelFS

The `sshare` command can be used to check your fairshare and \\(LevelFS\\) values. This command provides information about the resource usage and priority of users in the SLURM system.

```bash
$ sshare -Po user,fairshare,levelfs | grep $USER
abc123|0.456209|4.588063
```

In this example, `abc123` has a fairshare value of `0.456209` and a LevelFS value of `4.588063`.

A fairshare value of `0.456209` means a moderate amount of resources were used recently. A LevelFS value of `4.588063` indicates a moderate priority for scheduling jobs.