GPU Workload Observability: Metrics, Traces & Profiles

Requirements

GPU observability is not "Prometheus, but for GPUs." The workloads are bulk-synchronous distributed jobs: thousands of ranks march in lock-step and the slowest one sets the pace, so a per-host dashboard hides the real problem. The platform must let an engineer pivot from a fleet symptom (MFU dropped) to the guilty rank, to the guilty step, to the guilty CUDA kernel — and do it without adding meaningful overhead to the very jobs it watches.

Scale & estimates

Sizing for one large fleet: 100k GPUs = 12,500 nodes × 8 GPUs. The headline is that metrics are a firehose but tractable, traces are only affordable if you sample, and profiles are so large they never belong in the metrics path at all — they go to object storage and are fetched on demand.

Three pillars for GPU workloads

The classic metrics/traces/logs triad re-specializes for accelerators. Logs still exist, but the load- bearing pillars are metrics (is the fleet healthy and busy?), traces (which rank/step is slow, and is it compute or comm?), and profiles (which exact kernel, and why?). Each answers a different question at a different cost and frequency.

The pillars are linked by shared keys — job_id, rank, gpu_uuid, step — so a single click drills from an MFU dip (metric) to the slow step (trace) to the offending kernel (profile). Without those join keys you have three disconnected tools and a human doing the correlation by hand.

High-level design

On every node, lightweight exporters/agents emit metrics and trace spans; a profiler hook captures kernel timelines only when armed. Metrics and traces flow through a streaming ingest bus (Kafka) that decouples producers from stores and absorbs bursts, then fan out to a time-series DB (metrics) and a trace store. Profiles are far too large for the bus, so the hook writes them straight to object storage and only a small pointer (job/rank/step + URL) travels through ingest. A query + dashboard layer joins the three by shared keys, and an alerting path watches the streams for MFU drops, stragglers, and faults.

flowchart LR
    subgraph NODE["GPU Node (x12,500)"]
        DCGM["DCGM Exporter (metrics)"]
        OTEL["OTel Agent (trace spans)"]
        PROF["Profiler Hook (CUPTI/Nsight)"]
    end
    DCGM --> KAFKA["Streaming Ingest (Kafka)"]
    OTEL --> KAFKA
    PROF --> OBJ["Profile / Object Store"]
    KAFKA --> TSDB["Time-series DB (metrics)"]
    KAFKA --> TRACE["Trace Store"]
    KAFKA -->|pointer| OBJ
    TSDB --> QY["Query + Dashboards"]
    TRACE --> QY
    OBJ --> QY
    TSDB --> AL["Alerting"]
    TRACE --> AL
    QY --> ENG["SRE / ML Engineer"]
    AL --> ENG
      

• Streaming ingest decouples and protects. Kafka lets a node burst telemetry without stalling the workload, gives the stores back-pressure-free buffering, and provides a single place to sample, aggregate, and route. A storage hiccup never blocks the training loop.
• Three stores for three data shapes. A TSDB (columnar, delta-compressed) for metrics; a span-indexed trace store for distributed traces; cheap object storage for multi-GB profiles. Forcing all three into one backend is how teams accidentally build a petabyte metrics bill.
• Profiles bypass the hot path. Only a tiny descriptor flows through ingest; the bytes go node → object store directly. This keeps the pipeline cheap and makes "profile rank 12, steps 400-405" an O(few-GB) object write rather than a firehose.
• The query layer is the join engine. Drilling from metric → trace → profile works because every signal carries job/rank/gpu_uuid/step; the dashboard resolves those keys across the three stores on demand.

Deep dive: distributed tracing & profiling at scale

Tracing a single service is easy; tracing one training step across thousands of ranks is the hard problem. Every rank runs the same code at (almost) the same time, and the thing you care about is the skew between them. Three ideas make it work: a logical correlation key, aggressive sampling, and on-demand deep profiling.

Correlating a step across ranks

The correlation key is not wall-clock time — node clocks drift by milliseconds, which is an eternity at GPU speed. Instead, every rank tags its spans with (job_id, step) and the platform aligns them on the logical step boundary and on shared NCCL collective barriers, which are true synchronization points. Aligning on a barrier turns "30k noisy timelines" into a clean apples-to-apples comparison where the straggler pops out visually.

flowchart TD
    STEP["Training step N (all ranks)"] --> KEY["Correlation key = job + step N"]
    KEY --> SAMP{"Sample this step?"}
    SAMP -->|no| DROP["Metrics only (cheap path)"]
    SAMP -->|yes| SPAN["Per-rank spans: fwd, bwd, all-reduce"]
    SPAN --> ALIGN["Align on NCCL barrier (not wall clock)"]
    ALIGN --> VIEW["Timeline / flame view, slowest rank flagged"]
      

You cannot profile every step or every GPU

Full tracing is millions of spans per second and detailed profiling steals real GPU cycles, so capture is layered:

• Always-on metrics on every GPU — cheap, the baseline that triggers everything else.
• Head sampling of traces — emit step-level spans for rank 0 plus a small random set of ranks, and only on 1 of N steps. Enough to watch comm/compute ratio trend without the firehose.
• Tail / triggered sampling — when a step is anomalously slow or MFU dips, retain the spans for that window across all ranks so the straggler is never sampled away.
• On-demand deep profiling — kernel-level Nsight/PyTorch capture, armed manually or by an alert, for a handful of ranks and a handful of steps, uploaded to object storage.

On-demand deep profiling flow

A controller arms the profiler hook on the selected ranks for a bounded step window, each rank uploads its trace chunk, and the platform stitches and indexes the chunks by step so they line up into one aligned timeline (Perfetto / HTA) instead of N unrelated files.

sequenceDiagram
    participant U as ML Engineer
    participant C as Profiler Controller
    participant R0 as Rank 0
    participant Rk as Rank k
    participant S as Object Store
    U->>C: Deep profile job for 5 steps
    C->>R0: Enable profiler steps N..N+5
    C->>Rk: Enable profiler steps N..N+5
    R0-->>S: Upload trace chunk
    Rk-->>S: Upload trace chunk
    C->>S: Stitch and index by step
    S-->>U: Aligned timeline ready
      

The payoff is the flame/timeline view: per-rank kernel timelines stacked on a common step axis. A straggler shows as a rank whose compute lane is longer or whose all-reduce starts late; a data-starvation problem shows as GPU idle gaps waiting on the input pipeline; a comm problem shows as collectives dominating the step. Holistic Trace Analysis then aggregates many ranks into one verdict so you are not eyeballing 30k lanes.

Deep dive: controlling telemetry cost & cardinality

Observability that costs more than a few percent of compute is a failed design. Because metrics are always-on across 100k GPUs, the levers are about writing less, storing it cheaper as it ages, and refusing to explode cardinality — while still keeping the signals that fire the alerts that justify the whole platform.

Bottlenecks & scaling

Every part of this system has a failure mode that shows up only at fleet scale. The recurring theme: push work to the edge and to the right storage tier — pre-aggregate and sample at the agent/ingest layer, keep hot data small, and never let an unbounded label reach the TSDB.