Designing a Recommendation System
- Pradeep P
- 3 days ago
- 4 min read
Series: Modern System Design · Layer 6 — Modern systems
Layer 6 · Post 4 of 26
← Previous: Designing a Search System → Next: Designing a Feature Flag System
Layer 6 — Modern systems · Post 66 of 88
A recommendation system turns user behavior into ranked suggestions, under tight latency budgets and constantly changing catalogs.
What you'll learn
The two-stage shape: candidate generation (recall) then ranking (precision)
Why online serving is a feature lookup plus a small model, not "train on the request"
How feedback loops, cold start, and catalogs that change every minute break naive designs
The idea in one minute
You are not "running AI." You are building a retrieval and ranking pipeline with a latency budget (often 50–150 ms). Stage 1 pulls a few hundred items the user might like. Stage 2 scores them with more features and returns a dozen.
App / feed API | v [ Feature store: user + context ] ----> [ Ranker (model or rules) ] ^ ^ | | [ Candidate gens: collab / content / trending ]--+ ^ | [ Event log: views, clicks, hides ] --> [ Offline trainers ] | catalog store (items appear/disappear) | ranker timeout? fall back to trending / editorial
Training happens offline (or nearline). Serving reads precomputed embeddings, counters, and candidate sets.
Why it matters
Feeds, "because you watched," homepages, and "customers also bought" are the product. Interviewers listen for two stages, offline vs online, and what you do when the model is empty (new user, new item).
If you only describe matrix factorization on a whiteboard and skip serving, you have not designed a system.
How it works
Events. Clients emit views, clicks, purchases, dismisses. You collect them in a log (Kafka → warehouse). This is the training fuel and the online features ("clicked this category in the last 10 minutes").
Candidates. Multiple generators, unioned and deduped: collaborative (people like you), content (same tags/embeddings), popularity, and business constraints (in stock, not already purchased, region). Each generator is allowed to be approximate. Goal is recall.
Features. At request time you look up user features, item features, and context (time, device) from a feature store or caches. You do not scan the event log on the hot path.
Rank. A model (gradient boosted trees, a small neural net) or a well-tuned weighted sum scores candidates. You apply diversity rules so the list is not twelve of the same SKU. Goal is precision under a timeout.
Feedback. Impressions and clicks flow back. If you only train on clicks, you amplify whatever you already showed. Log impressions (what was shown) or you cannot reason about position bias.
Failure. Feature store miss: rank with fewer features or skip to popular. Model timeout: return candidate order. Catalog delete: filter tombstones at serving so you never recommend a 404.
Cold start: new users get geo/popular/onboarding picks until you have events. New items get content-based candidates until collaborative signals exist.
A simple example
You open a video app. The API loads your user embedding and recent watch IDs from Redis. Candidate services return 200 video IDs (similar to last watch, trending in your country, editorials). The ranker adds watch-time predictions and filters videos you already finished. You see 12 thumbnails in 80 ms. A brand-new user with no history gets "trending in India" plus the onboarding topics they tapped. A video published 30 seconds ago appears because a content generator used the title embedding, not because the collaborative model has seen it.
Common mistakes
One giant model over the whole catalog at request time. You cannot score 10 million items in 100 ms. Retrieve first.
Training on the serving path. GPUs and batch jobs live offline. The API loads weights or a compact scorer.
Ignoring the catalog. Recommendations of out-of-stock or unpublished items are bugs, not "ML."
Optimizing clicks only. You will rank clickbait. Pick an objective that matches the product (watch time, purchase, long-term retention) and say the tradeoff.
No fallback. Models fail. Trending and rules are production, not shame.
How this shows up in real systems
Netflix / YouTube / TikTok: many candidate sources, heavy ranking, ruthless latency budgets.
Amazon / Shopify "related products": item-to-item co-occurrence is often enough; you do not need a deep model to start.
Feature stores (Feast, Tecton, in-house): the glue between warehouse training and online lookup.
Search retrieved by query; recommendations retrieve by user. The next control plane is not ML — it is which code path is on: feature flags.
Recap
Candidates then rank, with features from a store, not from a live full-table scan.
Train offline; serve with caches and strict filters on the live catalog.
Design cold start and fallbacks or the first week of a new user is empty.
Shipping the ranker behind a flag is how you roll it out without a binary deploy. That is the next post.
Series: Modern System Design · Layer 6 — Modern systems
Layer 6 · Post 4 of 26
← Previous: Designing a Search System → Next: Designing a Feature Flag System
Comments