5,000,000 messages/second peak assumption
Hypothetical whole-cluster peak at 1,000 payload bytes/message. Average load is 500,000 messages/second. Neither rate is a benchmark.
InfrastructureStaff Level
Kafka Message Queue Tutorial & Distributed Log Design
Follow a conceptual Kafka example through partitions, committed offsets and a consumer crash, then size a hypothetical distributed log.
How does a Kafka message queue work?
In this conventional consumer-group model, producers append records to topic partitions. Each group tracks its own next offset. Processing before committing allows replay after a crash, so external actions need duplicate protection. Walk through offsets 40, 41 and 42 below. This is a conceptual tutorial; the official Quick Start in the sources provides runnable commands. Kafka also has share-group queue features, which are outside this model.
Worked Example
Kafka message-queue example with offsets and a crash
This conceptual walkthrough uses topic practice-events, partition 0, key session-7 and conventional consumer group practice-summary. Keep the partition count and assignment unchanged. Explicitly place all three events in partition 0. Its committed offset names the next record to read. Assume earlier records are complete, manual commits and retained records. No Kafka installation or runtime result is claimed.
Start at committed next offset 40
The partition contains offsets 40, 41, 42. Read practice-40 at offset 40.
Committed next offset remains 40. Fetching a record does not finish its external action.
Finish offset 40, then commit 41
Durably record a completed rehearsal in the practice database. Wait for success, then commit next offset 41.
Before commit: 40. After commit: 41. A restart now begins at 41.
Finish offset 41, then crash before committing 42
Apply practice-41 to add one practice point in the practice database. Crash after that database action succeeds, before the offset commit.
The practice point exists. Committed next offset is still 41; it never advanced to 42.
Restart the same group at offset 41
Group practice-summary resumes at 41 and receives practice-41 again.
A naive increment adds a second point. An idempotent destination atomically stores the event ID and the practice-point change together. On retry it recognizes that ID and skips the duplicate change. A separate marker write leaves another crash gap.
Resolve the retry, then continue to 42
After the destination confirms the action is already applied, commit 42. Process practice-42 to record a saved reflection, then commit 43.
Next offset moves from 41 to 42 to 43. The replayed event has one database effect only if the destination’s atomic deduplication contract holds.
Producer idempotence prevents retry duplicates in the Kafka log; it does not deduplicate consumer side effects. External databases and payments need destination cooperation. Other consumer groups have independent offsets. Ordering here is only within partition 0, never across the whole topic. For runnable setup, topic, producer and consumer commands, follow the official Apache Kafka Quick Start linked in the sources below.
Target Scale
Engineering Scale & Performance SLAs
Hypothetical interview sizing in decimal GB and TB. Validate throughput, latency and failure behavior on the chosen hardware, record sizes, batching, compression and security settings.
100 internal service clusters assumed
Service clusters are the modeled clients, not daily active users. Their traffic distribution is unspecified.
43.2 TB/day raw payload
Average raw payload before replication. 7 days across 3 copies require 907.2 TB of payload only; see the exclusions below.
Producer ack p99 < 10 ms target
Hypothetical producer-to-broker acknowledgement target, measured at the producer under stated load. Consumer processing and offset-commit deadlines are separate. No latency was measured here.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Append keyed practice events to topic partitions and preserve order within each partition.
- Track the next offset separately for each conventional consumer group and partition.
- Retain seven days of records in this hypothetical example; configure retention for the actual recovery window.
- Handle consumer retries with an explicit destination contract, as the offset example demonstrates.
Reliability & Latency SLAs
- Measure the assumed peak and producer acknowledgement target under representative load and failure conditions.
- Choose replication and minimum ISR settings deliberately; prefer rejected writes over silently weakening the chosen durability policy.
- Batch sequential I/O to reduce work. CPU, TLS and decompression still cost resources.
Stage 02
Capacity Estimation Math
Scope: a conventional consumer group reading retained partition logs. The classic replication example disables Eligible Leader Replicas (ELR). Version and feature settings can change recovery semantics.
Assumptions and measurement limits
Use replication factor 3, acks=all and min.insync.replicas=2. The producer waits for the current in-sync replica (ISR) set, which may contain two or three replicas. Below the minimum, writes can fail. This is not an unconditional three-ack rule.
Producer idempotence covers producer retry duplicates under its required configuration. It does not make an external consumer action exactly once.
Kafka read-process-write transactions can atomically commit output records and consumer offsets. Downstream read_committed consumers exclude aborted output. External sinks need their own coordination; an offset commit alone cannot make a database update or payment exactly once.
Capacity excludes compression effects, protocol overhead, indexes, segment overhead, operating headroom and consumer traffic. The seven-day retention assumption uses average traffic, not an all-day peak.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Hypothetical peak producer payload | 5,000,000 messages/second × 1,000 bytes/message (decimal 1 kB) | 5 GB/s producer payload; an assumed peak, not a measured or sustained guarantee |
| Peak follower replication payload | 5 GB/s × 2 follower copies | 10 GB/s broker-to-broker replication payload, excluding consumer traffic |
| Peak aggregate write payload | 5 GB/s × 3 total copies (one leader and two followers) | 15 GB/s across all copies; this is not internal network traffic |
| Average daily raw payload | 500,000 messages/second × 1,000 bytes × 86,400 seconds/day | 0.5 GB/s average = 43.2 TB/day before replication (decimal units) |
| Seven-day retained payload | 43.2 TB/day × 7 days × 3 total copies | 907.2 TB payload only; excludes compression effects, protocol overhead, indexes, segment overhead and operating headroom |
Sources for the example and design mechanisms
Reviewed October 23, 2026. Calculations use the hypothetical inputs above; source results are not benchmarks of this design.
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
Kafka Brokers · Partition Leaders · OS Page Cache
Accept appends and serve partition reads.
Follower Fetchers · ISR Tracking · High Watermark
Copy partition logs and track replication progress.
Group Coordinator · Offset Topic (__consumer_offsets) · Assignment Protocol
Coordinate membership and partition ownership; store committed next offsets, not proof of external action completion.
KRaft Controller Quorum · Broker Registry
Manage cluster metadata and partition leadership separately from data replication.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: (topic, partition_id, offset)
- topic
- partition_id
- offset
- key
- value
- timestamp
Conceptual record model, not a SQL table or Kafka file-format specification.
PK: (consumer_group, topic, partition_id)
- consumer_group
- topic
- partition_id
- committed_next_offset
This walkthrough starts at committed next offset 40 and ends at 43. The practice database owns external action deduplication.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Retained log and independent consumption
Option A: A conventional group shares partition ownership
Option B: Separate groups keep independent progress
Rationale: Choose group boundaries around the work. In this model a completed read does not delete a retained record. Share-group acknowledgement semantics require a separate design.
Data transfer and CPU budget
Option A: User-space buffering
Option B: Eligible file-to-socket transfer paths
Rationale: Some paths reduce copying. Kafka does not use sendfile with SSL enabled. Benchmark CPU and throughput with the actual encryption and compression configuration; no fixed context-switch count or network saturation is promised.
Technical FAQ
Frequently Asked Questions: Kafka Message Queue Tutorial & Distributed Log Design
Key interview questions and conceptual defenses.
How much storage does this Kafka example need?
At 500,000 messages/second and 1,000 bytes/message, the average payload is 0.5 GB/s or 43.2 TB/day raw. Retaining 7 days across 3 total copies requires 907.2 TB of payload. The 5,000,000 messages/second peak is a separate sizing assumption. Compression, protocol overhead, indexes, segment overhead, operating headroom and consumer traffic need separate budgets. These are hypothetical decimal units, not measured cluster capacity.
What happens when a partition leader crashes?
In this classic ELR-disabled example, a safe in-sync replica can take over. With acks=all and min.insync.replicas=2, writes can fail below the minimum. Data loss remains possible if all safe replicas are lost or unclean election permits an out-of-date leader. Check the deployed version and feature settings.
Why can offset 41 run twice after a restart?
The practice database applied the action, but the consumer crashed before committing next offset 42. The same group resumes at 41. Atomically recording the event ID with the database action can suppress its repeated effect. Producer idempotence alone does not protect this action.
Does Kafka run at memory speed with no CPU overhead?
No. Batching and sequential I/O reduce work, while page caching can avoid disk reads. TLS, compression and application processing still need resources. Hardware and workload measurements determine throughput and latency.
Simulate this architecture
Download ClawPad