5,000 Pages/sec
Sustained peak request volume during high-traffic events.
InfrastructureSenior Level
Distributed Web Crawler & URL Frontier
Crawl 10 billion web pages with politeness policies, deduplication, and distributed URL frontiers.
Target Scale
Engineering Scale & Performance SLAs
Target production parameters expected in a senior or staff interview round.
10 Billion Target URLs
Daily active users generating read and write operations.
500 TB compressed HTML
Projected data ingestion and replication storage capacity.
Throughput > 5k pages/s
Strict end-to-end percentile latency SLA constraint.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Download web pages across the public internet starting from seed URLs.
- Extract text content, metadata, and outgoing hyperlinks.
- Store parsed content and index documents for search engines.
Reliability & Latency SLAs
- Strict politeness (respect robots.txt and max 1 request/host/second).
- URL and content deduplication to prevent infinite crawl traps.
- Extensible document extraction pipeline (HTML, PDF, media).
Stage 02
Capacity Estimation Math
Step-by-step arithmetic conversions for QPS, storage, and bandwidth.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Crawl Velocity | 10 Billion pages ÷ 30 days = ~3,850 pages/sec | ~5,000 Peak Pages/sec |
| Storage Footprint | 10B pages × 50 KB compressed HTML | ~500 Terabytes raw storage |
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
Distributed Priority Queue · Host Politeness Delay Queues
Manages crawl scheduling, ensures domain rate-limiting, and balances URL freshness.
Asynchronous Chromium / Go Crawlers · HTML Parser & Link Extractor
Downloads pages, obeys robots.txt directives, and parses out new URLs and text tokens.
Bloom Filters / MinHash · Object Storage (S3 / Bigtable)
Filters seen URLs using Bloom filters and prevents near-duplicate content indexing with SimHash.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: url_hash
- url_hash (BINARY(32))
- original_url (TEXT)
- simhash (BIGINT)
- crawled_at (TIMESTAMP)
Stored in Bigtable / Cassandra keyed by reversed domain name.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Bloom Filter vs Database Index for Seen URLs
Option A: In-Memory Bloom Filter (Fast, 0.1% false positive, zero disk I/O)
Option B: Relational DB Unique Index (Exact, disk-heavy lookups)
Rationale: Bloom filters check billions of URLs in RAM in sub-microseconds without overloading disks.
Technical FAQ
Frequently Asked Questions: Distributed Web Crawler & URL Frontier
Key interview questions and conceptual defenses.
How do you prevent crawl traps (e.g. infinite calendar links)?
Implement maximum path depth limits, query parameter normalization, and domain-level URL crawl quotas.
Simulate this architecture
Download ClawPad