5,000,000 Messages/sec Peak
Sustained peak request volume during high-traffic events.
InfrastructureStaff Level
Distributed Message Queue (Apache Kafka)
Build a durable, append-only log message broker supporting high-throughput publish-subscribe and consumer group offsets.
Target Scale
Engineering Scale & Performance SLAs
Target production parameters expected in a senior or staff interview round.
100+ Internal Microservice Clusters
Daily active users generating read and write operations.
50 TB/day commit logs
Projected data ingestion and replication storage capacity.
Publish to consumer ack < 10ms
Strict end-to-end percentile latency SLA constraint.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Producers publish messages to named topics partitioned across broker nodes.
- Consumer groups consume messages independently with managed offset positions.
- Durable, configurable message retention (7 days by default, or compacted).
- Support at-least-once, at-most-once, and transactional exactly-once delivery semantics.
Reliability & Latency SLAs
- Massive throughput: millions of messages per second across commodity servers.
- High durability: zero message loss when leader broker crashes (In-Sync Replicas).
- Sequential disk I/O and zero-copy OS data transfers to eliminate CPU overhead.
Stage 02
Capacity Estimation Math
Step-by-step arithmetic conversions for QPS, storage, and bandwidth.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Peak Cluster Ingress Throughput | 5,000,000 msgs/sec * 1 KB average message size | 5.0 GB/second network & disk ingress |
| Replication Factor Overhead | 5.0 GB/s ingress * 3 replicas (1 leader + 2 followers) | 15.0 GB/second internal cluster bandwidth |
| 7-Day Storage Retention | 5 GB/s * 86,400s * 7 days * 3 replicas | 9.07 Petabytes total cluster raw storage |
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
Broker Fleet (Java/Rust/Scala) · Topic Partition Leaders · In-Memory Page Cache
Accept producer append requests, write sequentially to disk segments, and maintain in-sync replica (ISR) lists.
Follower Replica Fetcher · High Watermark Coordinator · Quorum Controller (KRaft)
Replicate commit log segments from partition leader and advance the High Watermark commit boundary.
Group Coordinator Broker · Offset Storage Topic (__consumer_offsets) · Rebalance Protocol
Assign topic partitions evenly to consumers in a group and track acknowledged commit offsets.
KRaft Controller Quorum (Raft) · Metadata Topic · Broker Registry
Manage topic schemas, partition reassignments, broker heartbeats, and leader failover elections.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: base_offset
- base_offset (UINT64)
- physical_position (UINT64)
- crc32 (UINT32)
- key_bytes (VARBINARY)
- value_bytes (VARBINARY)
- timestamp_ms (UINT64)
Appended sequentially to 1GB segment files on disk; binary sparse index maps offsets to physical file byte positions.
PK: (consumer_group, topic, partition_id)
- consumer_group (VARCHAR(128))
- topic (VARCHAR(128))
- partition_id (INT)
- committed_offset (BIGINT)
- commit_timestamp (TIMESTAMP)
Stored in internal compacted Kafka topic __consumer_offsets cached in broker memory.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Storage Architecture: Individual Message Queuing (RabbitMQ) vs Append-Only Log (Kafka)
Option A: Traditional Message Queue with individual message acknowledgment and deletion
Option B: Append-Only Commit Log with Consumer-Managed Offsets
Rationale: Deleting individual messages causes heavy random disk I/O and index thrashing. An append-only log uses sequential I/O and allows multiple independent consumer groups to read the same data at their own pace.
Network Transfer: User-Space Buffering vs Kernel Zero-Copy (`sendfile`)
Option A: Read from disk into user memory buffer, then write to socket
Option B: Zero-Copy `sendfile` syscall (OS Page Cache directly to Network Socket)
Rationale: Zero-copy eliminates 2 memory context switches and 2 CPU data copies per packet, allowing Kafka to saturate 10Gbps network cards with minimal CPU usage.
Technical FAQ
Frequently Asked Questions: Distributed Message Queue (Apache Kafka)
Key interview questions and conceptual defenses.
Why is Kafka so much faster on disk than traditional databases?
Kafka writes only sequentially to the end of segment files. Sequential disk access on modern NVMe drives (and even spinning HDDs) is nearly as fast as random memory access.
What happens when a partition leader broker crashes?
The KRaft controller selects a new leader from the In-Sync Replicas (ISR) list. Because all ISR followers have confirmed logs up to the High Watermark, zero messages are lost.
Simulate this architecture
Download ClawPad