Ace ML System Design & Data Architecture Loops.

Agree on value before choosing a model.

Use satisfied viewing sessions per active user as the business goal. For an initial measurable proxy, define a qualified watch as at least 30 seconds of a video lasting at least 60 seconds. Count sessions with at least one qualified watch, while checking hides, reports and satisfaction surveys. Watch time alone can reward content people regret watching.

Daily active users

1 million

Feed requests per user per day

10

Eligible video catalog

10 million

Server response target

p95 ≤ 200 ms

p95 means 95% of requests should complete within the target. Measure this over a stated window and traffic mix; it is a target here, not a measured result.

Rework these assumptions in the capacity math calculator. A 10× traffic assumption is a planning scenario, not evidence of the real peak.

Retrieve broadly. Rank a bounded set.

Scoring every video per request would mean up to 11.6 billion user–item scores/second at the assumed peak. Retrieving at most 1,000 candidates bounds that to about 1.16 million ranking scores/second before filtering. Those are item scores, not model requests; batch candidates per feed request.

Keep item embeddings and their approximate nearest-neighbor index in a serving tier. Batch-read user and item features from an online feature store. Cache a short-lived, per-user candidate pool; reapply current eligibility rules on every response. Never share a personalized cache entry across users.

Allocate an initial 20 ms for routing and context, 40 ms for retrieval, 30 ms for feature reads, 60 ms for ranking, and 20 ms for final filtering and serialization. This totals 170 ms, leaving 30 ms of the 200 ms deadline. These are engineering budgets, not independently measured percentiles; validate end-to-end tail latency under load.

Make training examples match serving reality.

Start with a popularity-by-language baseline. Then train a two-tower retrieval model: one encoder represents the viewer’s past activity, the other the video’s metadata and content. Train similar embeddings for qualified-watch pairs, with sampled negatives. A non-selected item is not necessarily disliked; compare negative-sampling choices on held-out retrieval quality.

Rank retrieved candidates with a gradient-boosted tree model using recent topic affinity, item age, historical quality and session context. Begin with binary cross-entropy for qualified-watch prediction. If negative examples are downsampled, weight training examples appropriately and check probability calibration on the natural exposure distribution. Add model complexity only when it beats the simpler baseline within the latency budget.

• Capture the decision. Log request ID, event ID, pseudonymous user ID, exposed item, position, event time and model version. Deduplicate retried events. Join outcomes only after a fixed label window, such as 24 hours; pending outcomes are not negatives.
• Prevent future information. Build each training row from features available at that impression’s time. A video’s eventual watch count or the viewer’s next click would leak the label. Recompute aggregates with time boundaries and fit preprocessing only on training data.
• Test the right time period. Train on earlier dates, validate on later dates and reserve the latest period for a final test. Also report new-user and new-item results; repeated users across time splits are not automatically leakage.
• Keep pipelines aligned. Reuse feature definitions, record freshness and compare served features with their offline reconstruction. Retrain daily initially; update recent-session features within a minute if measurements justify the cost. Version the retrieval encoders and index together.

Minimize retained identifiers and agree on retention and deletion rules. Propagate deleted users and removed videos into logs, features, indexes and future training inputs; do not keep them indefinitely just because they may help a model.

Offline gains earn an experiment, not a launch.

Historical logs reflect the old policy’s exposure and position bias. Offline metrics cannot establish the new policy’s causal effect. Reserve a small, bounded exploration stream among eligible items and log its selection probabilities; any bias correction needs sufficient overlap and checks for unstable weights.

Shadow traffic checks serving behavior without changing recommendations. A small canary checks operational safety. A properly powered experiment tests user benefit. None substitutes for the others, and there is no universal test duration without traffic and variance estimates.

A useful feed must survive missing information.

New viewer

Use chosen interests, language and a diverse popular feed. Adapt from session activity. Do not invent a personal history or require unnecessary sensitive attributes.

New video

Retrieve from content metadata before interaction history exists. Give eligible new items bounded exploration exposure and track coverage so popularity does not become permanent exclusion.

Feature or model timeout

Stop work at the deadline. Use a compatible model that was validated for missing features, or serve an eligible popular list. If current eligibility cannot be verified, omit the item, even if the feed becomes shorter.

Bad release or stale index

Monitor error rate, fallback rate, feature freshness, latency and outcome changes by segment. Roll back the model, feature schema and index as a compatible versioned set. Drift is a reason to investigate, not automatic permission to retrain and ship.

The central tradeoff. Approximate retrieval saves compute but can miss useful candidates; ranking cannot recover items it never sees. Larger candidate pools may improve recall while increasing feature traffic and latency. Start with the 1,000-candidate cap, measure both quality and cost, then change one constraint at a time.