Multi-Datacenter Model Serving with Latency SLAs

Requirements

Scope is the serving and routing layer across regions: training, the model registry internals, and single-replica inference mechanics are covered elsewhere (see LLM inference platform and model registry). Here the job is to take an already-deployed model and answer a global stream of requests within an SLA while GPUs are scarce and lumpy across regions.

Scale math

Concrete numbers keep the design honest. The table below is a plausible global LLM service; the exact figures matter less than the relationships — that one region holds most of the GPUs, and that a single cross-region hop can eat a large slice of the budget.

Global routing

Routing is the heart of the system, and it happens in two tiers. A global tier gets the request to the right region; a regional tier picks the right replica and decides whether to serve, overflow, or shed. The goal at every hop is the same: the lowest-latency replica that still has capacity, without violating the SLA.

Tier 1 — getting to a region (GeoDNS / anycast + L7)

• GeoDNS / anycast — DNS resolves (or anycast BGP routes) the user to the network-nearest edge by geography and measured latency. This is coarse and cache-friendly but blind to live GPU load; it answers “which region is close”, not “which region has a free GPU”.
• L7 global load balancer — at the edge, an application-aware proxy makes the real decision. It knows each region's health and a recent capacity signal (queue depth, free KV blocks) and can steer to the nearest region with capacity, not merely the nearest region.
• Why two layers — GeoDNS handles the 95% common case cheaply and is hard to change quickly (DNS TTLs); the L7 layer handles the fast-moving capacity decisions and overflow.

flowchart TD
    U["Users (global)"] --> DNS["GeoDNS / Anycast"]
    DNS --> GLB["Global L7 Load Balancer"]
    GLB --> GUS["US Gateway"]
    GLB --> GEU["EU Gateway"]
    GLB --> GAP["APAC Gateway"]
    GUS --> RUS["US GPU Replicas"]
    GEU --> REU["EU GPU Replicas"]
    GAP --> RAP["APAC GPU Replicas"]
    GAP -.->|overflow| GUS
    GEU -.->|overflow| GUS
    RUS --> KUS["KV Cache US"]
    REU --> KEU["KV Cache EU"]
    RAP --> KAP["KV Cache APAC"]
      

Tier 2 — choosing a replica inside a region

Once inside a region, a stateless gateway runs the admission decision. Three ideas dominate, in priority order:

• Load-aware, not round-robin — pick the replica with the most headroom by a live signal: queue_depth, free_kv_blocks, and in-flight decode slots. Round-robin ignores that one replica may be stuck on a long generation while another is idle.
• Session / prefix affinity — route a multi-turn session, or requests sharing a long system prompt, to the same replica so its KV cache is reused instead of recomputed. This can remove most of the prefill cost — a first-class latency win, not a micro-optimization.
• Overflow / spillover — if no local replica can meet the SLA, spill to the next-nearest region only if the extra RTT still fits the budget; otherwise queue briefly or shed with backpressure. Overflow must be bounded so a busy region cannot tip its neighbor over (see failover).

flowchart TD
    R["Request hits region gateway"] --> A{"Local capacity free?"}
    A -->|yes| B{"Affinity key present?"}
    B -->|hit| L["Sticky replica (reuse KV)"]
    B -->|miss| LL["Least-loaded local replica"]
    A -->|no| C{"Neighbor within budget?"}
    C -->|yes| O["Overflow to neighbor region"]
    C -->|no| D["Queue briefly, else shed (backpressure)"]
      

Capacity & autoscaling across regions

Because GPUs are scarce and unevenly placed, capacity planning is a global optimization, not a per-region one. Three levers interact: where models live, how each region scales, and what happens at the edge of capacity.

• Follow-the-sun capacity — peak load rotates with the work day. Rather than provision every region for its local peak, shift a shared GPU pool (or autoscale aggressively) toward whichever region is currently busy, and let overflow cover the seams.
• Autoscaling on the right signal — scale on queue depth and KV-cache pressure, not CPU. But GPU scale-up is slow (cold starts: pull ~140 GB of weights, warm the runtime), so keep a warm buffer and scale early.
• Queue vs shed vs degrade — at saturation you may queue (bounded, only if the SLA still allows), shed lowest-priority traffic, or degrade to a cheaper model. Never queue unbounded — that converts a capacity problem into a latency-SLA breach.
• Prioritization — protect paying / interactive traffic with priority classes; batch and best-effort work yields first when GPUs are tight.

Placement strategy — the core trade-off

Replicating a 140 GB model into every region buys the best latency but is the most expensive option, and it strands GPUs in low-traffic regions. The realistic designs sit between full replication and a single central pool.

Model & version consistency

Multiple regions serving “the same model” must agree on which version they run, or identical prompts return different answers depending on geography. Rollout is therefore a distributed, staged change — never a simultaneous global flip.

• Artifact distribution — the model registry is the source of truth: immutable, content-addressed weights pushed to regional object stores / CDN (or a P2P fan-out) before any region is told to switch. Weights are pre-staged; the cutover is just a config change.
• Canary, region by region — promote one region (often the smallest, e.g. APAC) to the new version, watch SLO and quality metrics, then expand. A bad version is contained to one region, not the planet.
• Consistent routing during rollout — while vN and vN+1 coexist, pin a session to one version so a user does not flip mid-conversation, and make overflow version-aware so a spilled request lands on a compatible replica.
• Config replication — routing tables, model aliases, and feature flags propagate through a versioned control plane with bounded staleness, so “gpt-x = vN+1” becomes true everywhere within a known window.

flowchart LR
    REG["Model Registry vN+1"] --> DIST["Artifact Distribution (CDN / P2P)"]
    DIST --> S1["Canary: one region"]
    S1 --> CHK{"SLO and quality OK?"}
    CHK -->|yes| S2["Expand to more regions"]
    S2 --> S3["Full fleet on vN+1"]
    CHK -->|no| RB["Rollback to vN"]
      

Failover & resilience

A region can fail outright (network partition, power, control-plane outage) or fail softly (GPUs saturated, latency climbing). The system must detect both, drain the bad region, and reroute — without tipping the rescuer into the same overload.

• Health checks & draining — continuous probes (liveness + a real inference canary) feed the global LB. An unhealthy region is drained: new traffic stops, in-flight requests finish, and the LB updates within seconds, not DNS-TTL minutes.
• Reroute to nearest healthy region — failed traffic shifts to the next-closest region with capacity. The extra RTT is now unavoidable, so the SLA may move to a documented degraded tier during the incident.
• Graceful degradation — if the rescuing region lacks GPUs for the full model, fall back to a smaller / quantized model, shorter max-tokens, or a cached answer. A correct, slightly weaker response beats a timeout.
• Avoid cascading overload — spillover is bounded by a rate limit and load shedding at the rescuer, plus circuit breakers so a sick region is not hammered by retries. Without caps, failover turns one region's outage into a global brownout.

sequenceDiagram
    participant U as User (APAC)
    participant G as Global LB
    participant A as APAC Region
    participant N as Neighbor (US)
    participant H as Health Checker
    H->>A: Probe healthz + canary
    A-->>H: Timeout (unhealthy)
    H->>G: Mark APAC drained
    U->>G: Inference request
    G->>N: Reroute to nearest healthy
    N->>N: Admit, maybe smaller model
    N-->>U: Response within degraded SLA
    Note over G,N: Bounded spillover + circuit breaker prevent cascade
      

Bottlenecks & scaling

The recurring failure modes all trace back to the same root: capacity is finite and geography is fixed. Naming them — with the mitigation — is the staff-level move.