Model Registry & Artifact Store

Requirements

We are building the control plane for models — the catalog that sits between training and serving. Training pipelines write finished models into it; CI/CD and the serving fleet read from it to decide what to run and to pull the bytes. Training itself, experiment tracking, and the inference servers are adjacent systems, not in scope here.

Scale & back-of-envelope

The numbers below show why metadata is the easy half. The catalog is megabytes of rows; the pain is entirely in moving immutable gigabytes to many nodes at once.

Takeaway: a single origin pull is fine; the simultaneous pull is what hurts. If every node fetches from the object store at once, demand (~3.1 TB/s) dwarfs any bucket’s egress, so the rollout serializes behind the origin and drags on for hours. The fix is to serve the bytes once and let them replicate — CDN edge caching plus peer-to-peer distribution — bringing fleet-wide completion back down to a small multiple of a single-node download (minutes, not hours).

Core entities

Five entities carry the model. The crucial design choice: a ModelVersion is immutable and points at an Artifact by content hash, while a Stage/Alias is a movable pointer. Promotion and rollback are therefore just alias updates — cheap, atomic, and auditable — never re-uploads.

A compact relational sketch — small, strongly-consistent, RDBMS-shaped:

Model
  model_id PK, name UNIQUE, owner, task_type, description, created_at

ModelVersion                       -- immutable; one row per registered version
  version_id PK, model_id FK -> Model,
  version_no,                       -- monotonic per model (1,2,3, ...)
  artifact_digest -> Artifact.digest,
  run_id FK -> Run,
  framework, metrics JSON, params JSON,
  status, created_by, created_at
  UNIQUE (model_id, version_no)

Artifact                           -- content-addressed blob / manifest
  digest PK,                        -- sha256 of the manifest (the model's identity)
  size_bytes, media_type,
  layers JSON,                      -- [ sha256 of each shared chunk/file ]
  storage_url, created_at

Run                                -- training provenance
  run_id PK, dataset_version, code_commit,   -- git sha
  container_image,                           -- image digest
  hyperparams JSON, started_at, finished_at

Alias                              -- movable promotion pointer
  alias, model_id FK -> Model,     -- PK (model_id, alias)
  current_version_id FK -> ModelVersion,
  updated_by, updated_at

API design

Writes (register, promote) are rare and gated; reads (resolve, get, download URL) are hot and cacheable. The download path hands back a presigned/expiring URL so bytes never flow through the registry service itself.

Register an immutable version

POST /v1/models/{name}/versions
{
  "run_id":          "run_8f1c...",
  "artifact_digest": "sha256:9f3a...",      // bytes already uploaded to blob store
  "framework":       "pytorch",
  "metrics":         { "eval_acc": 0.921, "f1": 0.88 },
  "params":          { "lr": 3e-4, "epochs": 3 }
}

201 Created  → { "version": 42, "version_id": "mv_...", "status": "registered" }
409 Conflict → this digest is already registered (idempotent re-register)

Get a specific version (immutable ⇒ cache forever)

GET /v1/models/{name}/versions/42
200 OK → { version, artifact_digest, framework, metrics, run_id, created_at, ... }
Cache-Control: public, max-age=31536000, immutable

Resolve an alias / stage (hot read path at deploy)

GET /v1/models/{name}/aliases/prod
200 OK → { "alias": "prod", "version": 42, "artifact_digest": "sha256:9f3a..." }
Cache-Control: public, max-age=10        // short TTL; invalidated on promote

Promote / set stage or alias (gated + audited)

PUT /v1/models/{name}/aliases/prod
{ "version": 42, "reason": "passed canary; +1.2% f1" }

200 OK        → alias now points at 42  (atomic; previous target retained for rollback)
403 Forbidden → caller lacks promote permission, or approval gate not satisfied

List / compare versions

GET /v1/models/{name}/versions?stage=staging&sort=-metrics.f1&limit=50
200 OK → { "versions": [ ... ], "next_cursor": "..." }

Get a download URL for the artifact (presigned, edge-served)

GET /v1/artifacts/{digest}/download?ttl=600
200 OK → {
  "url":        "https://cdn.example.net/blobs/sha256/9f/3a/9f3a...",
  "scheme":     "p2p+https",            // fall back to https if no peer mesh
  "expires_at": "2026-06-16T00:10:00Z",
  "size_bytes": 214748364800
}
# Client verifies sha256 of the downloaded bytes == digest before loading.

Rollback is not a special endpoint — it is just PUT .../aliases/prod { "version": 41 }. Because every prior version’s bytes still exist immutably, recovery is a pointer flip measured in milliseconds.

High-level design

Producers (training jobs, CI) write metadata + bytes; the registry service owns the strongly-consistent metadata DB and issues presigned URLs into the blob store. The serving fleet resolves an alias, then pulls bytes not from the registry but from a CDN + peer-distribution layer fronting the blob store — that is what survives the deploy fan-out.

flowchart LR
    TR["Training Jobs (register)"] --> REG["Registry Service (API)"]
    CI["CI / Promotion Pipeline"] --> REG
    REG --> MDB[("Metadata DB (versions, lineage, aliases)")]
    REG --> CACHE[("Resolve Cache")]
    REG --> BLOB["Blob / Object Store (write-once)"]
    BLOB --> CDN["CDN / Edge Cache"]
    BLOB --> P2P["Peer Distribution (Dragonfly / Kraken)"]
    CDN --> FLEET["Serving Fleet (1000s of nodes)"]
    P2P --> FLEET
    FLEET -->|resolve alias| REG
      

• Two stores, one hash. The metadata DB is small and consistent; the blob store is huge and immutable. A ModelVersion row glues them by artifact_digest, so the catalog can be replicated and cached freely while the bytes live wherever is cheapest.
• The registry is off the data path. Bytes flow client → CDN/peer mesh → client. The service only resolves metadata and mints presigned URLs, so a 200 GB pull never touches application servers.
• Resolve is cacheable. Version metadata is immutable (cache forever); alias resolution uses a short TTL plus invalidate-on-promote, so a rollout’s read storm is absorbed at the edge.
• Upload before register. Producers push bytes to the blob store first (content-addressed), then register the version referencing the digest — registration is metadata-only and fast, and re-registering the same digest is idempotent.

Deep dive: artifact storage, dedup & the fan-out problem

Artifacts are stored content-addressed: the key is the sha256 of the bytes. This gives three properties for free — immutability (the name is the content), integrity (the client re-hashes and verifies on pull), and dedup (identical bytes are stored once, no matter how many versions reference them).

Content-addressed dedup via layered manifests

A model is rarely one monolithic blob; it is a manifest of layers (base weights, adapters, tokenizer, config), each addressed by its own hash. Fine-tunes and re-exports share the heavy base layer, so a new version often costs only the bytes that actually changed:

# Manifest for ModelVersion 42  (sha256:9f3a...)
sha256:9f3a...  model-v42.manifest
  ├─ sha256:aaaa...  base_weights.safetensors   14.0 GB   ← shared by v40–v44
  ├─ sha256:bbbb...  lora_adapter.safetensors    120 MB   ← unique to v42
  ├─ sha256:cccc...  tokenizer.json              2.1 MB   ← shared
  └─ sha256:dddd...  config.json                   4 KB

# Registering v43 that only swaps the adapter stores ~120 MB, not 14 GB.

Because layers are immutable, a garbage collector can safely reclaim any blob no longer referenced by a manifest (mark-and-sweep over Artifact.layers), and cold versions can be tiered to cheaper/colder storage classes.

The deploy fan-out problem

Promoting a 200 GB model and rolling it to 1,000 nodes is 200 TB of egress in one window. Served from a single origin it is hopeless; the answer is to serve one copy and let it replicate. Edge seeders/super-peers warm from the origin, then nodes share chunks with each other so origin egress stays near one copy while fleet completion approaches a small multiple of a single-node download:

flowchart LR
    ORIG["Origin Object Store (serves ~1 copy)"] --> SEED["Edge Seeders / Super-peers"]
    SEED --> N1["Node 1"]
    SEED --> N2["Node 2"]
    N1 --> N3["Node 3"]
    N1 --> N4["Node 4"]
    N2 --> N5["Node 5"]
    N2 --> N6["Node 6"]
    N3 --> N7["Node 7"]
    N4 --> N8["Node 8"]
      

Deep dive: lineage, reproducibility & governance

A registry that only stores bytes is a liability; auditors and on-callers need to answer “exactly what is in prod, and can we rebuild it?” Lineage threads each version back to the run that made it, and that run back to its dataset version, code commit, and container image — a fully pinned recipe. Governance then guards the path from a registered version to prod.

flowchart LR
    DS["Dataset vN"] --> RUN["Training Run"]
    CODE["Code commit (git sha)"] --> RUN
    IMG["Container image digest"] --> RUN
    HP["Hyperparams"] --> RUN
    RUN --> MV["ModelVersion 42"]
    MV --> SIGN["Signed manifest (provenance)"]
    SIGN --> GATE{"Approval gate"}
    GATE -->|approved| PROD["alias: prod"]
    GATE -->|rejected| STG["stays in staging"]
      

• Reproducibility = pinned inputs. Storing the dataset version, the git SHA, the image digest, and the hyperparameters means a run can be re-executed to (near) bit-identical weights. Floating tags (:latest, mutable branches) break this — everything is referenced by immutable digest.
• Promotion gates. Moving to staging or prod can require passing evals, a canary, and a human/automated approval. The alias write is rejected unless the gate is satisfied, so policy is enforced at the system-of-record, not in a wiki.
• Audit trail. Every register and promote is an append-only event (who, when, from-version → to-version, reason). During an incident you can answer “what changed and who changed it” in seconds, and reconstruct the exact state at any past time.
• Signing & provenance (supply-chain security). Artifacts are signed (e.g. Sigstore-style) and carry attestations binding bytes to the build that produced them. Serving nodes verify the signature and digest before loading, so a tampered or unknown artifact is refused — defending against model poisoning and registry compromise.
• Access control. Read of a resolve/download URL, register, and promote are separately authorized; promotion to prod is the most privileged action and is always recorded.

Bottlenecks & scaling

A model registry almost never falls over on metadata QPS — the catalog is tiny. It degrades on bytes in motion and on a few hot keys during rollouts. The mitigations keep the consistent metadata core while pushing the heavy, immutable data out to caches and peers.