AI & Machine LearningPrincipal Level

High-Throughput LLM Inference Serving (vLLM)

Q: Does fitting the KV cache in memory prove how many requests I can serve?

No. In this hypothetical full-attention example, 4,096 cached prompt plus generated tokens use 10 GiB per sequence. An assumed 400 GiB cache budget fits 40 whole sequences, or 20 at 8,192 tokens. That is a memory ceiling, not a throughput, latency or admission guarantee. Reserve overhead, account for the actual KV-head configuration and cache placement, and benchmark your model and workload before setting request limits.

Q: How does speculative decoding speed up Large Language Model generation?

In a draft-model approach, a smaller model proposes candidate tokens and the target model verifies them, accepting a valid prefix. Not all proposed tokens will be accepted. Any speedup depends on acceptance rate, drafting and verification cost, model, traffic, hardware and sampling settings. Compare measured latency and throughput against ordinary decoding for the same workload.

Q: Why is network bandwidth between GPUs so critical for LLM serving?

Tensor parallelism partitions model computation and requires communication among GPUs. The cost depends on the parallel strategy, batch size and interconnect topology. Profile communication alongside compute on the chosen setup; a particular link name or bandwidth does not establish end-to-end latency.

Design a hypothetical large language model (LLM) service. The scale and latency figures below are interview targets, not measured ClawPad or vLLM results.

Practice in ClawPad Desktop ← All System Design Problems

Target Scale

Engineering Scale & Performance SLAs

Illustrative interview targets only. Validate feasibility for the selected model, hardware, prompt lengths and offered request rate; no benchmark is reported here.

Peak Throughput

50,000 Output Tokens/sec

Sustained peak request volume during high-traffic events.

Active Users

1 Million Daily AI Conversations

Daily active users generating read and write operations.

Storage Ingestion

1 TB/day prompt-response evaluation logs

Projected data ingestion and replication storage capacity.

Latency Budget

Time To First Token (TTFT) < 200ms

Strict end-to-end percentile latency SLA constraint.

Stage 01

Functional & Non-Functional Requirements

Establish clear problem boundaries before proposing architectural components.

Functional Scope

Core System Capabilities

Serve a selected open-weight model through a streaming API. Specify its exact configuration; parameter count alone does not define its attention or cache layout.
Streaming response tokens via Server-Sent Events (SSE) as they are generated.
Dynamic prompt prefix caching for multi-turn chat sessions and long system prompts.
Multi-GPU tensor parallelism and speculative decoding with smaller draft models.

Non-Functional Scope

Reliability & Latency SLAs

Illustrative targets: p99 time to first token (TTFT) under 200 ms and p99 inter-token latency (ITL) under 25 ms. Measure both under the stated workload before accepting these targets.
Maximum GPU High Bandwidth Memory (HBM) utilization without out-of-memory (OOM) crashes.
Horizontal autoscaling across GPU clusters with distributed request queuing.

Stage 02

Capacity Estimation Math

Hypothetical standard full-attention cache math, not a specification for every 70B, Llama or DeepSeek model. A key/value (KV) cache stores each layer's key and value vectors for tokens already processed.

Assumptions and measurement limits

Count KV heads, not query heads. Grouped-query attention lets multiple query heads share KV heads. Both examples use the stated KV-head count, a two-byte FP16 cache dtype (storage format), and cached prompt plus generated tokens. Sliding-window, compressed and other cache layouts need their own calculation.

GB means 1,000,000,000 bytes; GiB means 1,073,741,824 bytes. The assumed 400 GiB usable cache budget is after weights, runtime and other reserves. It is not a measured GPU-node capacity. The ceiling assumes independent sequences without prefix sharing.

A memory ceiling does not establish throughput, latency or admission capacity. Account for sharding, replication, per-device placement, block rounding and remaining runtime overhead before admitting requests.

To test the targets, record model/configuration and cache dtype, prompt/generated-length distribution, offered requests/sec, achieved output tokens/sec, TTFT and ITL percentiles, queued/rejected requests and cache occupancy. No serving benchmark was run for this example.

Dimension	Calculation Formula	Estimated Result
Ideal model weights only	70,000,000,000 parameters × 2 bytes/parameter (FP16/BF16)	140 GB before runtime, activations, cache and other overhead. This is not a GPU-count recommendation.
One sequence, 64 KV heads	2 (key + value) × 80 layers × 64 KV heads × 128 values/head × 4,096 cached tokens × 2 bytes/value	10,737,418,240 bytes = 10 GiB = 10.73741824 GB
Comparison, 8 KV heads	2 (key + value) × 80 layers × 8 KV heads × 128 values/head × 4,096 cached tokens × 2 bytes/value	1,342,177,280 bytes = 1.25 GiB = 1.34217728 GB
Memory ceiling at 4,096 tokens	floor(400 GiB usable cache ÷ 10 GiB/sequence)	40 whole sequences, each at 4,096 prompt + generated tokens
Memory ceiling at 8,192 tokens	8,192 / 4,096 × 10 GiB = 20 GiB/sequence; floor(400 GiB usable cache ÷ 20 GiB/sequence)	20 whole sequences at twice the cached context

Sources for the example and design mechanisms

Reviewed October 21, 2026. Calculations use the hypothetical inputs above; source results are not benchmarks of this design.

Stage 03

Multi-Tier Architecture & Component Topology

How requests navigate ingress gateways, application logic, caching, and persistence.

Inference Router & Load Balancer

KV-Cache Aware Router · Prefix Match Dispatcher · Fair Queuing Gateway

Prefer a node holding the same eligible token prefix when cache state and queue length justify it. Preserve isolation between requests and tenants when reusing cached state.

Inference Engine & Scheduler (vLLM / TensorRT-LLM)

Continuous Batching Scheduler · PagedAttention Virtual Memory Manager · Speculative Drafting Engine

Group active generation sequences into iteration-level batches; allocate non-contiguous physical GPU memory pages.

Model Execution & Tensor Parallelism Tier

Megatron-style Tensor Parallel Workers · NCCL GPU Communication · CUDA Kernels

Partition model computation across GPUs when needed. Communication adds overhead; profile the chosen interconnect and topology with the actual model and batch sizes before making latency promises.

Telemetry, Safety & Evaluation Tier

Guardrails Content Filter · Token Consumption Meter · Logfire / OpenTelemetry APM

Evaluate prompt safety, enforce customer token spend rate limits, and record TTFT/ITL latency percentiles.

Stage 04

Database Schemas & Partitioning Strategy

Entity models, indexing, and primary key partitioning.

Table: gpu_virtual_page_table

PK: (logical_token_index, physical_block_id)

logical_token_index (INT)
physical_block_id (INT)
gpu_device_id (INT)
reference_count (INT)

PagedAttention memory mapping table maintained in CPU host memory and synchronized with CUDA kernels.

Table: prompt_prefix_cache

PK: prefix_hash

prefix_hash (VARCHAR(64) PK)
node_id (VARCHAR(64))
token_length (INT)
last_accessed_at (TIMESTAMP)

Illustrative routing index, not the KV tensors themselves. Reuse depends on repeated exact eligible prefixes, routing and eviction; measure the hit rate for the workload.

Table: inference_requests_audit

PK: request_id

request_id (UUID PK)
model_name (VARCHAR(64))
prompt_tokens (INT)
completion_tokens (INT)
ttft_ms (FLOAT)
itl_ms (FLOAT)

ClickHouse analytics table for tracking model serving costs and throughput SLAs.

Stage 05

Critical Architectural Trade-Offs

How to defend engineering compromises when challenged by interviewers.

Decision Point

Batching Strategy: Static Request Batching vs Continuous Iteration-Level Batching

Option A: Static Batching (Wait for N requests, process together until longest finishes)

Option B: Continuous iteration-level batching (update the active batch between decoding iterations)

Rationale: A static batch can leave completed request slots unused while longer requests finish. Continuous batching lets finished requests leave and waiting requests join between iterations. Measure throughput and latency with the chosen model, hardware and request-length distribution; this mechanism has no fixed percentage gain.

Decision Point

KV Cache Allocation: Pre-allocated Contiguous Blocks vs PagedAttention

Option A: Pre-allocate contiguous VRAM for maximum context length (e.g. 8k tokens)

Option B: PagedAttention (allocate non-contiguous cache blocks as needed)

Rationale: Reserving maximum-length contiguous caches can leave memory unused. Block-based allocation can reduce unused allocation and enable sharing, but block rounding and shared-block lifetimes still affect usable memory. Measure occupancy and fragmentation for the configured workload; no universal waste percentage or concurrency multiplier follows.

Technical FAQ

Frequently Asked Questions: High-Throughput LLM Inference Serving (vLLM)

Key interview questions and conceptual defenses.

Does fitting the KV cache in memory prove how many requests I can serve?