GPU Cluster Scheduler / Resource Manager

Requirements

Treat the scheduler as a resource market: jobs bid for GPUs, the scheduler clears the market under priority, quota, and topology constraints while keeping utilization high. The hard part is not packing one job — it is packing gang jobs without deadlocking the cluster.

Scale & estimates

Numbers for a single large cluster. The point is to show that per-task scheduling throughput and cluster state size are both tractable on one host, so the scheduler can keep a consistent global view — which is exactly what gang placement needs.

Takeaway: the live state is small enough to hold in RAM on the scheduler, so a single active scheduler with a consistent snapshot can make global gang decisions; throughput is bursty, not sustained-huge, so the design optimizes for good packing decisions over raw decisions/sec.

Core entities

Six entities carry the whole model. A Job owns a gang of Tasks; Tasks consume GPUs on Nodes; Queues order jobs and carry policy; Quotas bound how much a tenant may take.

High-level design

The API admits jobs (quota/validation) into queues. A single active scheduler core runs a scheduling cycle: pick the next job by policy, ask the placement / bin-packer for a topology-fit set of GPUs, optionally invoke the preemption controller to free capacity, then the binder writes the assignment into the cluster state store. Node agents watch the store, launch containers on the assigned GPUs, and stream heartbeats/health back.

flowchart LR
    U["ML Engineer"] --> API["Scheduler API"]
    API --> Q["Queues (priority + quota)"]
    Q --> SCHED["Scheduler Core"]
    SCHED --> PLACE["Placement / Bin-packer"]
    SCHED --> PRE["Preemption Controller"]
    PLACE --> STATE["Cluster State Store"]
    PRE --> STATE
    STATE --> BIND["Binder"]
    BIND --> A1["Node Agent"]
    BIND --> A2["Node Agent"]
    A1 --> G1["8x GPU"]
    A2 --> G2["8x GPU"]
    A1 -->|heartbeat| STATE
    A2 -->|heartbeat| STATE
      

• Two-level / shared-state pattern. Like Omega/Kubernetes, the scheduler reads a consistent snapshot of free capacity, computes placements optimistically, and commits with a version check. This keeps the global view needed for gang decisions while allowing fast cycles.
• Binder is the source of truth, not the agent. A GPU is "taken" the moment the bind is committed to the state store (optimistic concurrency), so two cycles can't double-allocate.
• Node agents are reconcilers. They converge the node toward its assigned tasks and report actual state; the scheduler never RPCs GPUs directly.
• Preemption is a first-class subsystem. To admit a high-priority gang the scheduler may select victims (lower-priority/over-quota tasks), checkpoint or kill them, and reclaim their GPUs.

Deep dive: gang scheduling

Distributed training is bulk-synchronous: every rank must be alive to complete an all-reduce step. So tasks of a job must be scheduled all-or-nothing. If you instead allocate GPUs greedily one task at a time, you get the classic partial-allocation deadlock: two 64-GPU jobs each grab 50 of the 100 free GPUs, neither can reach 64, and both wait forever while 100 GPUs sit "used" but idle.

flowchart TD
    J["Gang job needs 64 GPUs"] --> CHK{"All 64 fit together?"}
    CHK -->|yes| RES["Reserve all tasks atomically"]
    CHK -->|no| HOLD["Hold in queue, no partial placement"]
    RES --> TOPO["Topology-aware bind (same NVLink island / rack)"]
    TOPO --> RUN["Launch all tasks together"]
    HOLD --> BACK["Backfill smaller jobs meanwhile"]
    BACK --> CHK
      

Three mechanisms make gang scheduling work in practice:

• All-or-nothing admission. The scheduler tentatively reserves a placement for every task; it commits only if the full set is satisfiable in one consistent snapshot. Partial reservations are released. (Kubernetes calls this a PodGroup; Volcano/Kueue and Slurm have direct gang/coscheduling support.)
• Topology-aware placement. All-reduce bandwidth dominates training throughput, so the binder prefers GPUs on the same NVLink island, then same rack/leaf switch, then same spine — minimizing cross-rack hops. A job split across the network can run 2-5× slower.
• Bin-packing. Pack jobs tightly onto fewest nodes/islands to leave large contiguous holes for future big gangs, rather than scattering and fragmenting the cluster.

The admission flow across nodes, including the all-or-nothing commit and rollback:

sequenceDiagram
    participant SC as Scheduler
    participant N1 as Node A
    participant N2 as Node B
    participant ST as State Store
    SC->>ST: Read free GPUs snapshot
    SC->>SC: Try to place 64-GPU gang
    alt All tasks fit
        SC->>N1: Reserve 32 GPUs
        SC->>N2: Reserve 32 GPUs
        SC->>ST: Commit gang all-or-nothing
        N1-->>SC: Tasks started
        N2-->>SC: Tasks started
    else Not enough free
        SC->>ST: Release tentative reservations
        SC->>SC: Requeue and age up priority
    end
      

The cost of all-or-nothing is wait: a big gang may sit idle while it accumulates enough free GPUs. Left unmanaged it both wastes capacity and head-of-line-blocks small jobs — which is exactly what reservations + backfill (next section) exist to fix.

Deep dive: fairness & quotas

Many teams share one cluster, so ordering can't be plain FIFO. Fairness has two layers: which job goes next (share/priority) and how much a tenant may hold (hierarchical quota), with preemption and backfill to enforce and optimize them.

• Dominant Resource Fairness (DRF). Jobs request multiple resources (GPU, CPU, RAM); DRF equalizes each user's dominant share — the largest fraction of any single resource they hold. On GPU clusters the dominant resource is almost always the GPU, so DRF effectively gives max-min fair GPU shares while still accounting for CPU/RAM-heavy jobs.
• Hierarchical queues. org → team → user, each with a weight and a guaranteed/max quota. Idle guaranteed capacity is lent to busy siblings (elastic), then reclaimed via preemption when the owner returns.
• Preemption. To admit a higher-priority or under-quota job, evict victims (lowest priority / most over-quota first). Prefer graceful preemption — signal the job to checkpoint then stop — over hard kill, so long training runs lose minimal work.
• Backfill. When a large gang is waiting for capacity, the scheduler makes a reservation for it at a future time, then lets smaller/shorter jobs run in the gaps as long as they finish before the reservation (EASY/conservative backfill). This raises utilization and stops a giant gang from starving everyone behind it.

The central tension: gang + bin-packing maximize the chance a huge job can ever run, but they raise wait times and fragmentation pressure. Backfill recovers the wasted capacity, and DRF + quotas + preemption keep the sharing fair while it happens.

Fault tolerance

At 100k GPUs something is always broken: nodes crash, GPUs throw XID/ECC errors, NICs flap. The scheduler must detect failures fast, recover gangs correctly, and never leave the cluster with stale or double-bound placements — all while the scheduler process itself can die.

sequenceDiagram
    participant FD as Failure Detector
    participant L as Scheduler Leader
    participant ST as State Store
    participant Q as Queue
    participant A as New Node Agent
    FD->>L: Node N heartbeat timed out
    L->>ST: Mark node N lost
    Note over L: A gang task ran on node N
    L->>L: Kill surviving sibling tasks
    L->>Q: Requeue the whole gang
    Q-->>L: Capacity freed elsewhere
    L->>A: Bind all tasks again
      

• Failure detection. Missed node heartbeats and GPU health signals (XID, ECC, NVLink down) mark a node degraded/lost. The failing GPU is cordoned so nothing new schedules onto it.
• Gang-aware recovery. Because training is all-or-nothing, losing one task usually kills the whole step. The scheduler tears down surviving siblings and requeues the entire gang (unless the framework supports elastic/fault-tolerant training). Checkpoint-restart on the job side caps lost work for week-long runs.
• Scheduler HA via leader election. Exactly one active scheduler at a time (active/standby) holds a lease in etcd/ZooKeeper. If it crashes, the lease expires and a standby takes over and rebuilds state from the store. A single writer avoids conflicting placement decisions.
• Avoiding stale placements. Binds use optimistic concurrency on a resource version: if free capacity changed since the snapshot, the commit fails and the cycle retries. A node agent rejects a bind for a GPU it already considers taken. Fencing tokens stop an old (paused) leader from committing once a new leader has taken over.
• Bin-pack vs spread, revisited. Packing maximizes contiguous capacity for big gangs but concentrates blast radius — one node failure can kill a tightly-packed gang. For replicated inference/services, prefer spread (anti-affinity) so a single node loss never takes down all replicas. Choose per workload class.

Bottlenecks & scaling

A GPU scheduler rarely falls over on raw QPS — it degrades through poor packing, head-of-line blocking, and contention on the global state. The mitigations keep the single-writer global view while removing the serial chokepoints.