Twitter / X Timeline

Requirements

Functional

• Create / edit / delete tweets; like, retweet, reply; search; follow users; account & login.
• View a home timeline of tweets from followed users.

Non-functional

• Read-heavy, low latency (home-timeline p99 ≤ ~200 ms); AP-leaning (eventual delivery OK).
• Horizontally scalable stateless services; durable systems of record; security (authN/Z, encryption, rate limiting, input validation).

Scale & back-of-the-envelope

• ~250M DAU; ~250M tweets/day (~3k/s avg, ~15–20k/s peak); read:write ~1000:1.
• Normal author (300 followers): 1 tweet = 300 timeline writes. Celebrity (100M): 1 tweet = 100M writes — a write storm → pull at read instead.
• Timeline cache stores tweet IDs only (~800/user) → tens of GB, sharded by user.

API design

POST   /v1/tweets            { text, mediaIds[] }   -> 201 { tweetId }
PATCH  /v1/tweets/{id}       { text }               -> 200
POST   /v1/tweets/{id}/replies | /likes | /retweets
POST   /v1/users/{id}/follow                        -> 200
GET    /v1/timeline?limit=20&cursor=<opaque>        -> { items[], nextCursor }
GET    /v1/search?q=...&cursor=<opaque>

Use opaque keyset cursors (createdAt + tweetId), never offset — offset re-scans skipped rows and breaks when new tweets arrive between pages.

High-level design

A CQRS-style system: the gateway forks into a write path (rate limiter → Tweet/Reply CRUD) and a read path (cache + Timeline Service). Tweet writes emit an event to a message queue → Timeline Fanout → Timeline Cache. CDC feeds Elasticsearch for search; media is served from the CDN.

flowchart LR
    Client["Web / Mobile"]
    CDN["CDN"]
    LB["Load Balancer L7"]
    GW["API Gateway"]
    RL{"Rate Limiter"}
    TweetSvc["Tweet CRUD Service"]
    TimelineSvc["Timeline Service"]
    SearchSvc["Search Service"]
    ProfileSvc["Profile Service"]
    Fanout["Timeline Fanout"]
    MQ["Message Queue"]
    TweetDB[("Tweet Content")]
    MediaDB[("Media Blob Store")]
    ES[("Elasticsearch")]
    TLCache[("Timeline Cache")]
    FollowDB[("Following Graph")]
    Client -->|HTTPS| LB --> GW
    CDN --> Client
    GW -->|write| RL --> TweetSvc
    GW -->|read| TimelineSvc
    GW --> SearchSvc
    GW --> ProfileSvc
    TweetSvc --> TweetDB
    TweetSvc --> MediaDB
    TweetSvc --> MQ --> Fanout -->|fanout on write| TLCache
    TweetDB -->|CDC| ES --> SearchSvc
    TimelineSvc --> TLCache
    TweetDB -->|fanout on read| TimelineSvc
    ProfileSvc --> FollowDB
    TweetDB --> CDN
    MediaDB --> CDN
      

Deep dive · hybrid fan-out

Normal authors → fan-out on write (push tweet IDs into each follower's Timeline Cache). Celebrities (followers above a threshold) → skip push to avoid a 100M-write burst; their tweets stay in Tweet Content and are pulled at read time and merged. The small per-celebrity result is cacheable, so millions of readers share one cached pull.

flowchart TD
    A["New tweet created"] --> B{"Author is celebrity?"}
    B -->|No| C["Fanout on write"]
    C --> D["Push tweet id to follower timelines"]
    B -->|Yes| F["Skip fanout, keep in Tweet Content"]
    H["Reader opens home timeline"] --> I["Read precomputed ids from cache"]
    I --> J{"Reader follows celebrities?"}
    J -->|Yes| K["Fanout on read celeb tweets"]
    J -->|No| L["Use cached ids only"]
    K --> M["Merge and rank"]
    L --> M
    M --> N["Return timeline page"]
      

Deep dive · write & read flows

The write is acked before fan-out completes; fan-out is async via the queue, so a celebrity burst never blocks the author's request.

sequenceDiagram
    actor U as User
    participant T as Tweet CRUD Service
    participant DB as Tweet Content
    participant MQ as Message Queue
    participant F as Timeline Fanout
    participant FG as Following Graph
    participant C as Timeline Cache
    U->>T: POST v1 tweets
    T->>DB: persist tweet
    T->>MQ: publish TweetCreated
    T-->>U: 201 Created
    MQ->>F: consume event
    F->>FG: get followers
    F->>C: push tweetId to timelines
      

sequenceDiagram
    actor U as User
    participant TL as Timeline Service
    participant C as Timeline Cache
    participant FG as Following Graph
    participant DB as Tweet Content
    U->>TL: GET v1 timeline
    TL->>C: read precomputed ids
    C-->>TL: tweet ids
    TL->>FG: get celebrity follows
    TL->>DB: fanout on read celeb tweets
    DB-->>TL: recent tweets
    TL->>DB: hydrate tweet ids
    TL-->>U: merged ranked page
      

Deep dive · Timeline Cache & hot keys

• Key timeline:{userId} → a capped, time-sorted list of tweet IDs only (Redis ZSET scored by createdAt) — small memory, no duplicated bodies; hydrate once from Tweet Content.
• Rebuildable derived view; on cold start/eviction, replay from source. Shard by userId.
• Hot keys (viral tweet / celebrity slice): don't push them; cache each celebrity's recent slice with short TTL; replicate hot shards; request coalescing (single-flight); CDN + read cache absorb repeats.

Data model

User Data (relational)   user_id PK, user_name, email, bio
Following Data (graph)   (:User)-[:FOLLOWS {since}]->(:User); per-user follower_count
Tweet Content (document) { tweetId, authorId, text, mediaIds[], createdAt, counts, isCelebrityAuthor }
Replies (document)       { replyId, tweetId, authorId, text, createdAt }
Media                    object store -> CDN
Timeline Cache           timeline:{userId} -> ZSET { tweetId : createdAt }, capped
Elasticsearch            { tweetId, authorId, text, createdAt }  kept fresh via CDC

Security

• AuthN/Z — Auth & IAM + token validation at the gateway; only the author edits/deletes their tweet.
• Encryption — TLS in transit, encryption at rest for all stores and media.
• Rate limiting — per-user/IP on write paths to stop spam/abuse.
• Input validation — sanitize tweet text, media types, query params (prevent injection/XSS).