CUDA Kernel Profiling & Auto-Tuning

Requirements

The system has one north-star goal: find and fix slow GPU kernels, and automatically choose the fastest kernel configuration for every shape and device the fleet cares about. "Fix" here is not hand-tuning PTX — it is steering an autotuner and then locking in the winner so the next thousand jobs inherit it for free. Two pressures shape every decision: profiling must be cheap enough to run on real workloads, and search must be cheap enough that we are not burning a GPU-week to shave a microsecond.

Profiling concepts

You cannot tune what you cannot name. A profiler dumps dozens of counters; the skill is mapping them to a single question: is this kernel limited by doing math (compute-bound) or by moving bytes (memory-bound)? Everything else — occupancy, stalls, the timeline — either supports that classification or tells you why you are not yet hitting the relevant ceiling.

The roofline model

The roofline is the mental model that ties it together. Every kernel has an arithmetic intensity — AI = FLOPs / bytes moved — and the hardware has two ceilings: peak compute (FLOP/s) and peak memory bandwidth (bytes/s). Plot achievable performance against AI and you get a roof with two slopes. Left of the ridge point a kernel is bandwidth-limited (raising AI is the only way up); right of it, it is compute-limited (you need more FLOP/s, e.g. tensor cores). Where your kernel sits versus the roof tells you both how far from optimal it is and which ceiling to push on.

flowchart TD
    K["Kernel"] --> AI["Arithmetic intensity = FLOPs / bytes"]
    AI --> CMP{"AI vs ridge point?"}
    CMP -->|low AI| MEM["Memory-bound: capped by HBM bandwidth"]
    CMP -->|high AI| CMPB["Compute-bound: capped by peak FLOPs"]
    MEM --> FIX1["Fix: coalesce, reuse, fuse to raise AI"]
    CMPB --> FIX2["Fix: occupancy, ILP, tensor cores"]
    FIX1 --> ROOF["Plot on roofline, compare to peak"]
    FIX2 --> ROOF
      

Two kernels can have identical runtime and need opposite fixes. A memory-bound softmax wants fusion and better reuse to raise AI; a compute-bound GEMM already near the bandwidth ceiling wants more occupancy and tensor-core utilization. The roofline stops you from "optimizing" the wrong dimension — adding FLOP/s to a memory-bound kernel buys nothing.

Profiling data pipeline

The naive approach — run Nsight Compute on production and read the report — does not survive contact with a fleet. Full kernel-metric replay can slow a kernel by 10-100× because the profiler re-runs each kernel many times to collect every counter. So the pipeline is layered by cost: cheap activity tracing is always on, expensive metric collection is sampled, and the heavy lifting (aggregation, ranking, regression) happens off the critical path in a store.

flowchart LR
    subgraph FLEET["Instrumented runs (many GPUs)"]
        APP["App + CUPTI / Nsight hook"]
        SAMP["Sampler: 1 of N runs"]
    end
    APP --> COL["Trace collector"]
    SAMP --> COL
    COL --> STORE["Profile store + aggregate"]
    STORE --> AN["Analysis: top kernels, regressions"]
    AN --> DASH["Dashboards + alerts"]
    DASH --> ENG["Performance engineer"]
    ENG -->|drill in| STORE
      

• Instrument with CUPTI / Nsight. The CUPTI Activity API is the workhorse: it streams kernel launch records (name, grid/block, duration, stream) with low overhead, no replay. Reserve Nsight Compute-style metric collection (occupancy, stall reasons, bandwidth) for sampled deep dives, because it replays kernels and is expensive. Two tiers, two overhead budgets.
• Collect traces from many runs / GPUs. A node-local collector batches and compresses activity records and ships pointers to a central pipeline. Sampling keeps overhead low: trace 1-in-N runs, or arm deep metric collection only for a few iterations on a few ranks. The same kernel shows up across thousands of runs, so you do not need every run to characterize it.
• Store + aggregate. Raw traces are large and write-once → object storage. The aggregate — per (kernel, shape, gpu, build) stats like p50/p95 duration, call count, total time, classification — goes to a queryable store. Aggregation is where "300M launch records" becomes "the 20 kernels that own 80% of the time."
• Surface top kernels and regressions. Analysis ranks kernels by total fleet time (your tuning backlog, in priority order) and diffs each build against a baseline to flag slowdowns. The output is two short lists — tune these and something got slower — not a wall of counters.

Deep dive: auto-tuning the kernel config space

This is the core of the system. A single logical op — say a GEMM of a given (M, N, K) — can be compiled into thousands of distinct kernels that all compute the same result but at wildly different speeds. The knobs are things like tile sizes (BLOCK_M/N/K), block / warp dims, unroll factors, pipeline depth (software pipelining / multi-stage shared-memory prefetch), vector width, and num_warps / num_stages. The best combination depends on the shape and the GPU, and no closed-form rule picks it — so we search. This is exactly what the Triton @autotune decorator, CUTLASS profilers, and TVM/Ansor do.

The compile-and-benchmark loop

The atomic operation is: take a candidate config, compile the kernel variant, benchmark it on the target GPU (warm up, lock clocks, time many iterations, take a robust statistic), and keep the fastest. Compilation is not free and benchmarking burns real GPU time, so the loop is wrapped in caching and smart search to run it as few times as possible.

sequenceDiagram
    participant T as Autotuner
    participant Ca as Config cache
    participant K as Compiler
    participant G as Benchmark GPU
    T->>Ca: Lookup best config for shape + GPU
    Ca-->>T: Cache miss
    loop Each candidate config
        T->>K: Compile kernel variant
        K-->>T: Binary
        T->>G: Benchmark, timed run
        G-->>T: Latency and throughput
    end
    T->>Ca: Store fastest config
    Ca-->>T: Cached for reuse
      

Search strategies

The config space is combinatorial — easily 10^3 to 10^6 points once you cross all the knobs — and most points are invalid (exceed shared memory or register budgets) or slow. The strategy is how you avoid benchmarking all of them:

In practice these compose: a cost model proposes, an evolutionary search explores around its predictions, and only the top handful are measured on hardware — because the measured benchmark is the ground truth and also the expensive part.

Caching best configs per shape + GPU

Tuning is amortized by never tuning the same thing twice. The output of a tune is a row in a config cache keyed by (op, dtype, shape-bucket, gpu_arch, toolkit_version) → winning config + measured time. A few things make the cache effective:

• Shape bucketing. Tuning every exact shape is hopeless, so shapes are bucketed (powers of two, padded to tile multiples). A new shape that hashes to a known bucket is an instant cache hit.
• Key on the GPU and toolkit. A config tuned for an A100 is often wrong on an H100, and a compiler upgrade can shift the winner — so arch and version are part of the key, never assumed.
• Persist and share. The cache is a fleet-wide artifact (object store + small index), so one tune on one GPU benefits every subsequent job. Cold start tunes; warm start reads.

Performance regression detection

Correctness has CI; performance usually does not — which is why kernels silently get slower. A code refactor, a compiler bump, or a new driver can knock 20% off a GEMM and no test goes red. This system closes that gap with a performance CI: a curated benchmark suite, per-kernel timing baselines, and a gate that flags slowdowns across commits, compiler, and driver versions.

Cross-reference with observability. Regression detection and the profiling pipeline are two views of the same data. CI catches a slowdown before merge in a controlled harness; fleet observability catches the slowdowns CI missed — a shape CI never benchmarked, a driver rolled out to production, a kernel hot only under real traffic. A regression flagged in CI deep-links to the same per-kernel profile drill-down the observability stack uses, so the responder starts at the evidence, not a red checkmark.

Bottlenecks & scaling

Every part of this system has a failure mode that only shows up at fleet scale, and they all trace back to the same tension: real measurement is expensive (it perturbs workloads and burns GPU time), so the design is a series of moves to measure less while learning more — sample, cache, model, and bucket.