50,000 Output Tokens/sec
Sustained peak request volume during high-traffic events.
AI & Machine LearningPrincipal Level
High-Throughput LLM Inference Serving (vLLM)
Serve generative Large Language Models with continuous batching, PagedAttention KV cache memory management, and speculative decoding.
Target Scale
Engineering Scale & Performance SLAs
Target production parameters expected in a senior or staff interview round.
1 Million Daily AI Conversations
Daily active users generating read and write operations.
1 TB/day prompt-response evaluation logs
Projected data ingestion and replication storage capacity.
Time To First Token (TTFT) < 200ms
Strict end-to-end percentile latency SLA constraint.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Serve 70B+ parameter open weights models (Llama 3, DeepSeek) over standard OpenAI-compatible REST/gRPC streaming APIs.
- Streaming response tokens via Server-Sent Events (SSE) as they are generated.
- Dynamic prompt prefix caching for multi-turn chat sessions and long system prompts.
- Multi-GPU tensor parallelism and speculative decoding with smaller draft models.
Reliability & Latency SLAs
- Time To First Token (TTFT) p99 under 200ms; Inter-Token Latency (ITL) under 25ms.
- Maximum GPU High Bandwidth Memory (HBM) utilization without out-of-memory (OOM) crashes.
- Horizontal autoscaling across GPU clusters with distributed request queuing.
Stage 02
Capacity Estimation Math
Step-by-step arithmetic conversions for QPS, storage, and bandwidth.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Model Weights GPU VRAM Footprint | 70 Billion parameters * 2 bytes (FP16/BF16) = 140 GB (or 70 GB in INT8/FP8) | Requires 2x 80GB NVIDIA H100 GPUs minimum for weights alone |
| KV Cache Memory per Request | 2 * layers(80) * heads(64) * dim(128) * tokens(4096) * 2 bytes (FP16) | 1.07 GB VRAM per active 4k context sequence |
| Concurrent Concurrency on 8x H100 Node | (640 GB total VRAM - 140 GB weights) / 1.07 GB per sequence | ~467 concurrent streaming requests per 8-GPU node |
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
KV-Cache Aware Router · Prefix Match Dispatcher · Fair Queuing Gateway
Inspect user system prompt hash and route requests to the GPU node that already holds cached KV prefixes.
Continuous Batching Scheduler · PagedAttention Virtual Memory Manager · Speculative Drafting Engine
Group active generation sequences into iteration-level batches; allocate non-contiguous physical GPU memory pages.
Megatron-style Tensor Parallel Workers · NCCL NVLink High-Speed Interconnect · CUDA Kernels
Split matrix multiplications across 8 GPUs simultaneously via NVLink (900 GB/s) for sub-20ms forward passes.
Guardrails Content Filter · Token Consumption Meter · Logfire / OpenTelemetry APM
Evaluate prompt safety, enforce customer token spend rate limits, and record TTFT/ITL latency percentiles.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: (logical_token_index, physical_block_id)
- logical_token_index (INT)
- physical_block_id (INT)
- gpu_device_id (INT)
- reference_count (INT)
PagedAttention memory mapping table maintained in CPU host memory and synchronized with CUDA kernels.
PK: prefix_hash
- prefix_hash (VARCHAR(64) PK)
- node_id (VARCHAR(64))
- token_length (INT)
- last_accessed_at (TIMESTAMP)
Redis cluster index enabling router to achieve 80%+ cache hits on common developer system prompts.
PK: request_id
- request_id (UUID PK)
- model_name (VARCHAR(64))
- prompt_tokens (INT)
- completion_tokens (INT)
- ttft_ms (FLOAT)
- itl_ms (FLOAT)
ClickHouse analytics table for tracking model serving costs and throughput SLAs.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Batching Strategy: Static Request Batching vs Continuous Iteration-Level Batching
Option A: Static Batching (Wait for N requests, process together until longest finishes)
Option B: Continuous Iteration-Level Batching (Cellular batching on each forward token pass)
Rationale: Static batching causes severe GPU idle starvation because short requests wait for long requests to finish. Continuous batching inserts new requests on every token step, increasing throughput by 400%.
KV Cache Allocation: Pre-allocated Contiguous Blocks vs PagedAttention
Option A: Pre-allocate contiguous VRAM for maximum context length (e.g. 8k tokens)
Option B: PagedAttention (Allocate small 16-token non-contiguous memory blocks on demand)
Rationale: Pre-allocating contiguous memory wastes 60-80% of GPU VRAM due to internal and external fragmentation. PagedAttention mirrors OS virtual memory, reducing memory waste to under 4% and doubling concurrency.
Technical FAQ
Frequently Asked Questions: High-Throughput LLM Inference Serving (vLLM)
Key interview questions and conceptual defenses.
How does speculative decoding speed up Large Language Model generation?
A tiny draft model (e.g. 1B params) rapidly generates 5 candidate tokens at high speed. The large 70B model verifies all 5 tokens in a single parallel forward pass, achieving 2x-3x faster generation.
Why is network bandwidth between GPUs so critical for LLM serving?
Tensor parallelism splits attention matrix multiplications across GPUs. Between layers, GPUs must exchange partial activations via `AllReduce`. Without high-speed NVLink (900 GB/s), network transfer bottlenecks the entire pipeline.
Simulate this architecture
Download ClawPad