Facebook News Feed

Requirements

Functional

• Create posts; follow users; view a personalized feed; page through it. (Likes/comments, privacy are bonus.)

Non-functional

• Eventual consistency, ~1 minute — a new post may take up to ~1 min to appear; this makes async fan-out acceptable.
• ~500 ms latency for posting and viewing; 2B DAU, read-dominated, AP-leaning, multi-region.

Scale & back-of-the-envelope

The 100:1 ratio justifies fan-out on write (pay once at write so each read is one cheap query). The celebrity case breaks naive fan-out — the problem the hybrid design solves. Feed storage is bounded by capping each user's feed (~1,000 newest) ≈ 100 TB with TTL.

API design

POST /posts                  { content }                 -> 201 { postId }
PUT  /users/{id}/followers   { }   # idempotent follow   -> 200
GET  /feed?pageSize&cursor    -> { posts[], nextCursor }   # cursor-based

creatorId/userId derive from the auth token, never from the body.

High-level design

Stateless services behind a gateway, backed by DynamoDB, plus an async fan-out pipeline. The write path (async push) returns 201 immediately; the read path (≤500 ms) reads the precomputed feed merged with a live pull of celebrity posts.

flowchart LR
    Client["Client App"] --> GW["API Gateway"]
    GW --> PS["Post Service"]
    GW --> FS["Feed Service"]
    GW --> FoS["Follow Service"]
    PS --> Posts[("Posts Table")]
    PS -. "publish event" .-> Q(["Post Create Queue"])
    Q -. consume .-> FW["Feed Workers"]
    FW -. "read followers" .-> FGSI[("Follow GSI")]
    FW -. "write entries" .-> PF[("PrecomputedFeed")]
    FS --> PF
    FS --> PostGSI[("Post GSI")]
    FS --> Posts
    FoS --> Follow[("Follow Table")]
      

Deep dive · fan-out on write vs read

Fan-out on read (pull): at read time, query who you follow, then their recent posts via the Post GSI (creatorId PK, createdAt SK), merge by time. Trivial writes, but a 1,000-followee user triggers a 1,000-way scatter-gather on every open — impossible under 500 ms at 230k+ reads/s.

Fan-out on write (push): an async pipeline precomputes each follower's feed via the Follow GSI (reverse index "who follows this creator?").

sequenceDiagram
    actor U as User
    participant PS as Post Service
    participant Q as Post Create Queue
    participant FW as Feed Workers
    participant FG as Follow GSI
    participant PF as PrecomputedFeed
    U->>PS: POST /posts
    PS->>Q: publish postId creatorId
    PS-->>U: 201 postId
    Q->>FW: deliver event
    FW->>FG: query followers of creatorId
    FG-->>FW: follower list
    loop each follower
        FW->>PF: put userId createdAt postId
    end
      

Deep dive · the celebrity problem + hybrid

A 100M-follower post fans out to 100M writes and its Follow GSI row is a giant hot partition. Solution — a precomputed flag on each follow edge: normal creators are pushed; celebrities are precomputed=false (skip fan-out) and pulled at read time. The Feed Service merges both.

flowchart TD
    A["New post by creator"] --> B{"Follower count high?"}
    B -- "No normal user" --> C["Push to PrecomputedFeed"]
    B -- "Yes celebrity" --> D["Skip fan-out, edge precomputed=false"]
    C --> E["Read = single PrecomputedFeed query"]
    D --> F["Pull at read via Post GSI"]
    E --> G["Feed Service merges both sources"]
    F --> G
      

Celebrities are few but huge (pull a handful, cache it); normal creators are many but modest (push, keep reads O(1)). The flag flips when a creator crosses a follower threshold. Fan-out is idempotent (keyed by (userId, createdAt, postId)) so queue redelivery is safe.

Deep dive · ranking, caching & pagination

• Ranking: two-stage — precomputed candidate generation (cheap) + ranking at read time on the small candidate set (EdgeRank-style affinity × content_weight × time_decay + ML), using a cached feature store to fit 500 ms.
• Caching: PrecomputedFeed is itself a materialized cache; a Redis hot-feed cache holds the first page for active users; immutable Posts rows + celebrity posts cache aggressively; media via CDN.
• Pagination: cursor/keyset on (createdAt, postId) — stable as new posts prepend; the cursor encodes the k-way merge watermark across precomputed + celebrity streams.

Data model (DynamoDB)

Posts        PK id; content, creatorId, createdAt
Post GSI     PK creatorId, SK createdAt        # recent posts by a creator (pull)
Follow       PK userFollowing, SK userFollowed; createdAt, precomputed   # hybrid flag
Follow GSI   PK userFollowed, SK userFollowing # "who follows this creator?" (fan-out)
PrecomputedFeed  PK userId, SK createdAt; postId   # cursor key; trim/TTL to ~1000