GPU Cluster Network Topology

Requirements

Most system-design problems are dominated by storage or request/response latency. A training fabric is different: the workload is one giant bulk-synchronous job where thousands of GPUs march in lockstep. After every micro-step they exchange gradients and activations through collective communication — all-reduce, all-gather, reduce-scatter, all-to-all. Those primitives are all-to-all-ish traffic patterns that stress the whole bisection of the network at once, and they are synchronization barriers: the step cannot advance until the slowest rank finishes. That is why the network, not the matrix multiply, is the design problem.

Compute grows with model FLOPs; communication grows with model size divided by link bandwidth. As you scale out, the second term wins — a poorly built network leaves tens of thousands of GPUs idle waiting on an all-reduce. The goal of the fabric is to make communication cheap, uniform, and predictable enough that collectives hide behind compute.

Functional requirements

• Connect 10,000+ nodes (~80k–100k GPUs) into a single logical fabric that any rank can reach any other rank.
• Efficiently support the collective primitives NCCL relies on (all-reduce, all-gather, reduce-scatter, broadcast, all-to-all) — not just point-to-point reachability.
• Provide RDMA with GPUDirect so a NIC can DMA straight into remote GPU HBM, bypassing the kernel and CPU.
• Expose the physical topology to the scheduler and to NCCL so jobs can be placed and collectives ringed along the fastest links.

Non-functional requirements

Bandwidth hierarchy

There is no single “network speed” in a GPU cluster — there is a steep hierarchy, and every tier you descend costs roughly an order of magnitude of bandwidth and adds latency. Understanding this cliff is the foundation of every placement and topology decision: keep the chattiest traffic on the fastest tier, and cross the slow tiers as rarely as possible.

The drop from NVLink (~900 GB/s) to a single 400 Gb/s NIC (~50 GB/s) is roughly 18×. That gap is the cost cliff: bandwidth inside a node is plentiful and (relatively) cheap; bandwidth between nodes is scarce and expensive, because it is built from optical transceivers and switch ports. This is the difference between scale-up — putting more GPUs into one tightly-coupled NVLink domain (e.g. an NVL72 rack of 72 GPUs) — and scale-out, adding more nodes across the InfiniBand / Ethernet fabric. Scale-up gives you the best-connected GPUs you can buy; scale-out gives you count, at the price of crossing the slow tier.

A single GPU node is itself a small network. The diagram below shows the three tiers that matter for design: GPUs talk to each other through an NVSwitch at NVLink speed, each node reaches the fabric through one or more RDMA NICs, and those NICs climb a switching hierarchy of leaf (top-of-rack) and spine (core) switches.

flowchart TD
  subgraph ND["GPU node (8x GPU)"]
    G0["GPU 0"]
    G1["GPU 1"]
    G7["GPU 7"]
    SW["NVSwitch (~900 GB/s)"]
    NIC["RDMA NIC (~400 Gb/s)"]
  end
  G0 --> SW
  G1 --> SW
  G7 --> SW
  SW --> NIC
  L1["Leaf switch (ToR)"]
  L2["Leaf switch (ToR)"]
  S1["Spine switch"]
  S2["Spine switch"]
  NIC --> L1
  L1 --> S1
  L1 --> S2
  L2 --> S1
  L2 --> S2
      

The bandwidth hierarchy in one picture: HBM inside each GPU, NVLink/NVSwitch within the node (~900 GB/s), then a step down to RDMA NICs (~400 Gb/s) that climb leaf and spine switches to reach the rest of the cluster.

Topologies: fat-tree vs dragonfly vs torus

Once traffic leaves the node it enters a switching topology, and the choice of topology fixes the cluster’s bisection bandwidth, its worst-case hop count (diameter), its cost, and how far it scales. Three families dominate the conversation.

Fat-tree (folded Clos)

A fat-tree is a multi-tier Clos network: nodes hang off leaf (top-of-rack) switches, leaves connect up to a spine tier, and large clusters add a third core tier. If each switch dedicates equal bandwidth up and down (1:1), the fabric is non-blocking and delivers full bisection bandwidth — any node can reach any other at line rate. Latency is uniform because every leaf-to-leaf path is the same small number of hops (3–5). The price is hardware: a non-blocking Clos needs a lot of switches and a lot of long optical cables, and the spine grows with the cluster. This is the default for GPU training clusters, almost always in a rail-optimized variant.

flowchart TD
  subgraph SPINE["Spine tier (core)"]
    S1["Spine 1"]
    S2["Spine 2"]
  end
  subgraph LEAF["Leaf tier (ToR)"]
    L1["Leaf 1"]
    L2["Leaf 2"]
    L3["Leaf 3"]
  end
  S1 --> L1
  S1 --> L2
  S1 --> L3
  S2 --> L1
  S2 --> L2
  S2 --> L3
  L1 --> R1["Rack 1 nodes"]
  L2 --> R2["Rack 2 nodes"]
  L3 --> R3["Rack 3 nodes"]
      

A non-blocking fat-tree: every leaf connects to every spine. Because all leaf-to-leaf paths traverse the same two hops through the spine, latency is uniform and the bisection is full.

Dragonfly

A dragonfly arranges routers into groups that are fully connected internally, with a sparse mesh of global (long, expensive optical) links between groups. The win is cost at extreme scale: it needs far fewer long-haul cables than a Clos while keeping the diameter tiny (~3 global hops). The catch is that those global links are oversubscribed, so a naive shortest-path route creates hotspots; dragonfly therefore requires adaptive routing (e.g. UGAL, which may send a packet the “long way” through an intermediate group to dodge congestion), and latency is non-uniform. Common in HPC fabrics (Cray Slingshot).

Torus

A torus wires each node directly to its neighbors in a 2D/3D mesh whose edges wrap around. With few or no switches it is the cheapest to cable and scales wiring linearly, which is excellent for nearest-neighbor communication. But its diameter grows with the cube root of node count, so distant pairs suffer many hops and high latency, and global all-reduce at huge scale is poor unless the collective is mapped carefully to the mesh. Google TPU pods use a 3D torus, augmented with optical circuit switches for reconfiguration and fault isolation.

Topology-aware collectives & placement

A fast fabric is necessary but not sufficient: the collective library and the scheduler have to map the communication pattern onto the physical wires. NCCL builds rings and trees across the detected topology. A ring all-reduce is bandwidth-optimal — each GPU sends and receives only 2(N-1)/N of the data — and is preferred for the large gradient buffers of data-parallel training. A tree all-reduce is latency-optimal (logarithmic depth) and wins for small messages or very large scale. NCCL auto-detects NVLink, PCIe, and NICs and lays its channels along the fastest links.

Rail alignment

The dominant GPU-cluster layout is the rail-optimized fat-tree. Each of a node’s 8 GPUs has its own NIC wired to a distinct rail — a dedicated leaf switch. All the GPU-0s across every node share rail 0, all the GPU-1s share rail 1, and so on. The payoff: a collective among same-rank GPUs stays within a single rail (one leaf hop, no spine crossing). Cross-rail reshuffles are handled inside the node over NVLink. The result is a hierarchical collective — reduce-scatter on NVLink, all-reduce across the rail over the fabric, all-gather back on NVLink — that keeps almost all bytes on the fastest available tier.

flowchart TD
  subgraph NA["Node A"]
    A0["GPU 0"]
    A1["GPU 1"]
  end
  subgraph NB["Node B"]
    B0["GPU 0"]
    B1["GPU 1"]
  end
  R0["Rail 0 leaf switch"]
  R1["Rail 1 leaf switch"]
  A0 --> R0
  B0 --> R0
  A1 --> R1
  B1 --> R1
  A0 --- A1
  B0 --- B1
      

Rail-optimized fabric: same-index GPUs share a rail (leaf switch), so their all-reduce never touches the spine. NVLink (the dashed intra-node links) handles cross-rail shuffles, keeping bytes on the fast tier.

Placement and congestion

Because oversubscription bites only when traffic crosses tiers, where a job lands is a first-class performance decision. A topology-aware scheduler co-locates each data-parallel all-reduce group on adjacent leaves within one island and aligns ranks to rails; a job scattered across the fabric pays repeated spine crossings and stalls on every collective. Even on a non-blocking fabric, flow collisions hurt: ECMP hashes a few elephant flows onto the same uplink, leaving other links idle. Adaptive routing (per-packet or flowlet spraying) spreads those flows, and in-network reduction (SHARP) performs the sum inside the switch, roughly halving the bytes on the wire. The scheduling side of this story — gang placement and topology-aware bin-packing — is covered in the GPU cluster scheduler.

RDMA & congestion control

Moving gradients at 400 Gb/s with the CPU in the loop is impossible — the copies and interrupts alone would dominate. RDMA (Remote Direct Memory Access) lets a NIC read and write remote memory directly, with kernel bypass and zero copy, for microsecond latency at near-zero CPU cost. GPUDirect RDMA goes one step further: the NIC DMAs straight into and out of GPU HBM, skipping the bounce through host memory entirely. Two fabrics deliver it.

RDMA NICs historically tolerate packet loss badly (a single drop can trigger expensive retransmission and crater throughput), so the fabric must be effectively lossless. InfiniBand achieves this natively with credit-based flow control. RoCEv2 must build losslessness on Ethernet: PFC (Priority Flow Control) pauses a traffic class before a switch buffer overflows, while ECN marks packets under load and DCQCN reacts by throttling the sender’s rate to keep queues short. The art is keeping queues shallow enough that PFC rarely fires — because PFC pauses cause head-of-line blocking and can spread congestion (even deadlock) across the fabric if mistuned.

Fault tolerance

At 10,000+ nodes the fabric contains hundreds of thousands of optical links, and optics are the single largest source of failures. With that many components, something is always broken — a link flaps, a transceiver degrades, a switch dies. The fabric must survive this as routine, not as an incident, because a stalled fabric idles the entire (very expensive) job.

• Path redundancy. A multi-tier fat-tree offers many equal-cost paths (every leaf reaches every spine). Losing one spine or one uplink doesn’t cut connectivity — it shaves bisection bandwidth (lose 1 of 8 spines ≈ a 12.5% capacity hit), degrading gracefully instead of failing.
• Adaptive rerouting. A fabric manager (the InfiniBand subnet manager or an Ethernet controller) detects a link-down event, recomputes routes, and reprograms switch forwarding tables. Adaptive routing steers individual flows around the dead or congested link without restarting the job.
• Gray failures. The dangerous case isn’t a clean death — it’s a flaky optic with rising CRC / symbol-error counters that quietly slows one link. Because collectives run at the speed of the slowest rank, one sick link becomes a cluster-wide straggler. Detect it via per-link error counters and per-rank step-time, then quarantine the hardware before it tanks the job.
• Draining a rail. When a leaf/rail switch degrades past tolerance, drain it: stop scheduling onto it, migrate or restart the affected ranks on healthy hardware, and flag it for repair. Paired with gang scheduling and checkpoint/restore, the job resumes from its last checkpoint on a clean island.
• Bounded blast radius. A core-switch failure touches far more traffic than a leaf. Build the fabric so the unit of failure is contained to a pod/island, and let the scheduler route new jobs around drained zones.

sequenceDiagram
  participant N as Node NIC
  participant L as Leaf switch
  participant FM as Fabric manager
  participant SCH as Scheduler
  N->>L: send collective on rail 2
  Note over L: spine uplink fails (CRC errors)
  L->>FM: report link down
  FM->>FM: recompute routes (adaptive)
  FM-->>N: reroute via healthy spine
  Note over FM,SCH: errors persist on the rail
  FM->>SCH: drain rail 2, mark node suspect
  SCH->>SCH: restart ranks on healthy island
      

Detect, reroute, drain: a failing uplink is rerouted in-fabric for transient errors; if errors persist, the rail is drained and the scheduler restarts affected ranks from the last checkpoint on a healthy island.

Bottlenecks & scaling

Scaling an interconnect from one rack to 10k+ nodes is a march from one bottleneck to the next. Each one shows up as a dip in MFU, and each mitigation trades cost, complexity, or bandwidth for predictability.