12,000 Writes / 350,000 Reads Peak
Sustained peak request volume during high-traffic events.
High-ThroughputSenior Level
Twitter / X Timeline & Fan-Out Engine
Design a high-throughput microblogging feed with fan-out-on-write vs fan-out-on-read for celebrity accounts.
Target Scale
Engineering Scale & Performance SLAs
Target production parameters expected in a senior or staff interview round.
300 Million DAU
Daily active users generating read and write operations.
250 GB/day text & metadata
Projected data ingestion and replication storage capacity.
Home timeline p99 < 50ms
Strict end-to-end percentile latency SLA constraint.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Post a tweet (280 chars) with media attachment IDs.
- View personalized home timeline of tweets from followed users.
- View user profile timeline in chronological order.
- Search tweets by keyword and hashtag in real time.
Reliability & Latency SLAs
- Sub-50ms p99 latency for home timeline retrieval.
- High availability (99.99%) over immediate strict consistency (eventual consistency acceptable).
- Tolerate celebrity accounts with 100M+ followers without cascading worker backpressure.
Stage 02
Capacity Estimation Math
Step-by-step arithmetic conversions for QPS, storage, and bandwidth.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Peak Tweet Ingestion QPS | 500M tweets / 86,400s = 5,787 avg QPS * 2x peak factor | 11,574 peak write QPS |
| Home Timeline Read QPS | 300M DAU * 5 visits/day / 86,400s * 20 page refreshes | 347,222 read QPS |
| Daily Tweet Text & Metadata Storage | 500M tweets * 500 bytes metadata/text payload | 250 GB/day storage growth |
| Active Timeline Memory Cache | 30M daily active users * 800 tweet IDs * 8 bytes (6.4 KB per user) | 192 GB RAM Redis Cluster |
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
BGP Anycast DNS · Envoy Reverse Proxy · API Gateway
TLS termination, OAuth2/JWT verification, token bucket rate limiting, and request routing.
Tweet Service · Snowflake ID Generator · Distributed Document DB (Cassandra/ScyllaDB)
Validate tweet payloads, mint chronological 64-bit IDs, persist to append-only tables, and publish to Kafka.
Kafka Fan-out Topic · Fan-out Consumer Fleet · Redis Cluster Timeline Cache
Push tweet IDs to active followers' timeline caches for standard users; merge celebrity tweets dynamically on read.
Kafka Search Stream · Earlybird Lucene Indexer · Elasticsearch Cluster
Real-time tokenized inverted indexing and trending hashtag aggregation with sliding window counters.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: tweet_id
- tweet_id (BIGINT PK)
- author_id (BIGINT)
- content (VARCHAR(280))
- media_ids (JSONB)
- created_at (TIMESTAMP)
Clustered on Snowflake tweet_id for natural time-ordered distribution across partition shards.
PK: (follower_id, followee_id)
- follower_id (BIGINT)
- followee_id (BIGINT)
- created_at (TIMESTAMP)
Secondary index on followee_id for rapid fan-out worker recipient resolution.
PK: user_id
- user_id (BIGINT PK)
- tweet_ids (REDIS ZSET)
- last_refreshed_at (TIMESTAMP)
Redis Sorted Set scored by tweet_id timestamp capped at 800 items per user.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Fan-out on Write (Push) vs Fan-out on Read (Pull)
Option A: Pure Fan-out on Write (Push to all followers on every tweet)
Option B: Hybrid Model (Push for accounts < 25k followers; Pull on read for celebrities)
Rationale: Celebrities with 100M+ followers cause millions of writes per second that saturate queues. A hybrid model precomputes timelines for ordinary users while merging celebrity tweets dynamically at read time.
Timeline Storage: Full Tweets vs Tweet IDs
Option A: Cache entire tweet JSON payload in Redis timeline
Option B: Cache only 8-byte Snowflake Tweet IDs in Redis and hydrate via multi-get
Rationale: Caching full payloads consumes tens of terabytes of expensive RAM. Caching 800 64-bit integers takes under 7 KB per user and hydrates in <2ms via batched multi-key gets.
Technical FAQ
Frequently Asked Questions: Twitter / X Timeline & Fan-Out Engine
Key interview questions and conceptual defenses.
How does Twitter handle users who have not logged in for 30 days?
Inactive users are dropped from active Redis timeline caches. When they log in again, their timeline is generated on demand via fan-out on read.
Why use Snowflake IDs instead of UUIDv4 for tweets?
Snowflake IDs encode a 41-bit millisecond timestamp, allowing database indices and Redis ZSETs to remain naturally sorted chronologically without secondary index lookups.
Simulate this architecture
Download ClawPad