Design a High-Throughput Timeline & Feed Generation Engine (Twitter / X / Instagram)
A timeline generation system balances high write amplification against fast sub-50ms reads by pushing tweets to followers of regular accounts, while pulling and merging celebrity tweets on-demand.
Live Architecture Studio
Edit components, modify labels, add databases, or redraw connections directly on this canvas:
Loading Twitter Timeline & Feed Blueprint...
Mounting vector diagram elements, nodes, and capacity metrics
Loading Twitter Timeline & Feed Blueprint...
Mounting vector diagram elements, nodes, and capacity metrics
1. Problem & Challenge
When a standard user with 200 followers tweets, pushing the tweet ID into 200 inbox caches takes 5ms. But when a celebrity with 100M followers tweets, pure push requires 100 million writes, overloading queues and lagging delivery.
2. Core Building Blocks & Responsibilities
👉 Desliza la tabla para ver roles y responsabilidades| Component | Role | Plain-English Explanation |
|---|---|---|
| Tweet Ingestion & Snowflake ID | Post Ingress | Stamps 64-bit k-ordered Snowflake IDs so tweets across distributed database shards are naturally chronologically sortable. |
| Fanout Decision Worker | Hybrid Fork | Checks author follower count. If <25,000 followers, fans out to followers Redis timelines. If >=25,000, writes only to author User Timeline. |
| Redis Timeline Cache | In-Memory Inbox | Stores the top 800 tweet IDs as a sorted set for active users. Storing only 8-byte IDs reduces timeline RAM usage by 95%. |
| Blender & Home Mixer | Fanout-on-Read Merger | When a user opens Twitter, fetches their pre-computed Redis timeline, pulls followed celebrity tweets, and runs a fast k-way merge sort. |
| Memcached Hydration Cluster | Entity Fetcher | Hydrates the top 20 merged tweet IDs with full JSON text, author avatars, and like counts in a single batch MGET in <5ms. |
3. Step-by-Step Request Flow
User Posts Tweet
Client sends POST /tweets. Server writes tweet to Manhattan/MySQL and emits event to Kafka.
Fanout Service Evaluates Author
Worker queries FlockDB social graph. If standard account, pushes tweet_id to all follower Redis lists.
User Requests Home Timeline
Home Mixer fetches cached Redis tweet IDs and pulls recent tweets from followed celebrity accounts.
K-Way Merge & Hydration
Blends lists by Snowflake timestamp and batch-hydrates text and media from Memcached.
4. Architectural Trade-offs
Pure Push vs Pure Pull vs Hybrid Fanout
Chosen: Hybrid Fanout Strategy
Rationale: Pure push causes catastrophic write amplification for celebrities (100M writes per tweet). Pure pull collapses under 500K read QPS. Hybrid gives sub-50ms reads for 99% of users without queue congestion.
Storing Full Tweet Content vs Tweet IDs in Cache
Chosen: Storing 64-Bit Tweet IDs in Cache
Rationale: If a tweet is deleted or edited, storing IDs means only 1 Memcached record needs updating, rather than mutating millions of pre-computed follower Redis timelines.
Interview Tip
Explain the Inactive User TTL Eviction: Never fan out tweets to users who have not opened the app in 30 days. Let their cache expire. If they return after 6 months, lazily rebuild their timeline on read from the database.
Explore Related System Blueprints
TinyURL Shortener
A URL shortener converts a long link (like a 100-character article URL) into a compact 7-character key (like tinyurl.com/xyz123) and redirects visitors in under 15 milliseconds.
API Rate Limiter
A rate limiter acts as a digital bouncer at the door of your API, ensuring each client stays within their allowed request limits (e.g. 100 requests per minute) and blocking abusive traffic.
Video Streaming CDN
Streaming high-definition video to millions of smart TVs and mobile phones requires breaking large 10GB video files into tiny 5-second chunks, encoding each into 20 different resolutions, and caching them right inside local ISP networks.
Uber Dispatch Engine
A real-time geospatial dispatch system matches riders with the most optimal nearby drivers using 64-bit H3 hexagonal indexing and 2-second batch optimization, minimizing city-wide pickup ETA and driver idle time.
Stripe Payments Ledger
A resilient financial payments architecture guarantees strict consistency (CP system) using cryptographic idempotency reservation, double-entry balanced postings, and sharded balance locks.
Figma Multiplayer Engine
A real-time multiplayer document engine uses stateful sticky session routing and server-authoritative operational ordering to sync 2D scene graphs across worldwide collaborators without locking.