10,000,000 Data Points/sec
Sustained peak request volume during high-traffic events.
InfrastructureStaff Level
Distributed Time-Series Metrics & Alerting Pipeline
Collect, aggregate, and alert on 100 million metric data points per second with sub-minute query SLAs.
Target Scale
Engineering Scale & Performance SLAs
Target production parameters expected in a senior or staff interview round.
50,000 Monitored Hosts
Daily active users generating read and write operations.
1 TB / day time-series data
Projected data ingestion and replication storage capacity.
Query p99 < 200ms
Strict end-to-end percentile latency SLA constraint.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Collect high-frequency metrics (CPU, memory, QPS, custom gauges) from server agents.
- Support multi-dimensional label querying (e.g. env=prod, region=us-east).
- Trigger alerting rules and PagerDuty escalations within 30 seconds of an anomaly.
Reliability & Latency SLAs
- High write ingestion throughput with zero data drop.
- Fast range queries over multi-day time windows.
- Automatic downsampling and data retention tiering (Hot -> Warm -> Cold).
Stage 02
Capacity Estimation Math
Step-by-step arithmetic conversions for QPS, storage, and bandwidth.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Ingestion Volume | 50,000 hosts × 2,000 metrics/host ÷ 10s collection interval | ~10,000,000 metric data points / sec |
| Storage with Gorilla Compression | 10M points/sec × 1.37 bytes/point (Gorilla XOR time delta encoding) | ~13.7 MB/s write rate = ~1.18 TB / day |
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
OTel / Prometheus Agent · Kafka Metrics Bus
Buffers high-frequency metric streams and routes by metric name hash.
VictoriaMetrics / ClickHouse / M3DB · Gorilla Compression Block
Compacts time-series chunks in 2-hour in-memory blocks before flushing to columnar disk storage.
PromQL Query Engine · AlertManager
Evaluates alerting rules every 15 seconds against in-memory sliding metric buffers.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: (series_id, timestamp)
- series_id (BIGINT)
- timestamp (TIMESTAMP)
- value (DOUBLE)
- labels_json (JSONB)
Columnar storage compressed with double-delta timestamp encoding and Gorilla float XOR.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Push vs Pull Metric Collection
Option A: Pull Model (Prometheus scrapes endpoints at intervals)
Option B: Push Model (Agents stream metrics to central gateway)
Rationale: Hybrid model: Pull for internal long-lived Kubernetes pods; Push for ephemeral serverless functions.
Technical FAQ
Frequently Asked Questions: Distributed Time-Series Metrics & Alerting Pipeline
Key interview questions and conceptual defenses.
How does Gorilla compression reduce time-series storage by 90%?
Gorilla compresses timestamps using delta-of-deltas encoding and float values using XOR with the previous reading, reducing average size from 16 bytes down to 1.37 bytes.
Simulate this architecture
Download ClawPad