01A feature gets defined — how many times, and by whom?
Exactly once: a feature is registered as entity, source, and transformation, and that single definition materializes into BOTH the offline store for training and the online store for serving.
Premium preview
ML Feature Store
Design an ML feature store that serves the same features to both offline model training and low-latency online inference for a company like Uber.
00
Practice checkpoints
The requirements are open as a taste. From the numbers onward, the full guide opens in the app.
- 01Clarify scope
- 02Requirements + scale
- 03API + data modelUnlocks in the app
- 04Draw architectureUnlocks in the app
- 05Deep diveUnlocks in the app
- 06Trade-off decisionUnlocks in the app
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02What do training and serving actually call?
Two reads, one truth: get_historical_features returns point-in-time-correct training frames from the warehouse; get_online_features returns the latest values for an entity in single-digit milliseconds.
03Can a training row see a value computed after its own event?
Never: training joins are as-of the label’s event timestamp — each row sees the newest feature value at or before that moment, so nothing computed after the prediction time leaks into the model.
04Team B wants a feature Team A already built — how do they find it?
Through the registry: every feature carries a searchable definition, an owner, a freshness SLA, and a consumers list, so reuse is the default and duplicate half-matching features are the exception.
05A feature ships today — can models train on last year of it?
Yes, via backfill: the same registered transformation replays over the historical event log to populate offline history, so a new feature is trainable immediately, not after a year of accumulation.
06How fresh is fresh — one answer for every feature?
No — each feature declares a freshness tier: daily batch for slow-moving facts, streaming within seconds for behavioral counters, and on-demand computation for values that only exist at request time.
Out of scopeModel training infrastructure and experiment tracking (the store feeds trainers, it is not one) · Embedding similarity search (that is the vector database question) · Label creation and data labeling pipelines (the store joins labels, it does not produce them)
Non-functional requirements
01How fast must an online lookup be?
Single-digit milliseconds at p99 for a batched multi-get — the model’s whole inference budget is often ~100 ms, and feature fetch is only one slice of it.
02How close must the served value be to the trained value?
Identical by construction: one transformation definition feeds both stores, so training-serving skew cannot be introduced by a second implementation — the classic silent model killer.
03A materialization job stalls — who notices, and when?
Monitoring does, before the models do: every feature carries a freshness SLA, and lag, null rates, and distribution drift alert operators while the damage is still upstream.
04How heavy can training reads get?
Very — point-in-time joins scan billions of historical rows, so the offline store is columnar and partitioned by time: throughput-shaped, where the online store is latency-shaped.
05The online store misses a key — does inference block?
No: a miss returns the feature’s declared default and the request proceeds; the default is versioned and its hit rate monitored, because a default is a model input, not an error path.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- Which models consume this store, and what is the strictest online latency budget among them?
- Is point-in-time-correct training data a hard requirement, or do these models tolerate approximate joins?
- What freshness does each feature family actually need — seconds, minutes, or daily is fine?
- How many teams will share features, and who owns a definition when two of them want it to change differently?
- What are the entities — users, merchants, trips — and roughly what cardinality for each?
01
Unlock the full playbook for ML Feature Store
Numbers, architecture diagram, API and data model, deep dives, expected topics, self-check, whiteboard starter, and common mistakes unlock inside the app.
Numbers that force architecture decisions
Locked in the app
Architecture path
Locked in the app
API and data model
Locked in the app
Deep dive directions
Locked in the app
+Series