ML Model Serving Platform

01What kinds of models does this platform serve?

General-purpose prediction serving — ranking, vision, fraud/tabular — any framework (XGBoost, PyTorch, TF) behind one uniform predict API on a standard runtime; LLM token streaming is a different platform with different physics.

02How does a team ship a new model version?

Push the artifact to the registry with a version, a schema signature, and training metadata; deploying is promoting that registry entry — never rebuilding a service image around the weights.

03Do requests arrive with features, or does the platform fetch them?

Both: callers may send raw features, but the common case is entity ids — the platform multi-gets the rest from the feature store using the same feature definitions training used.

04What stands between a new version and live traffic?

A rollout ladder: shadow (mirror traffic, serve nothing), then a canary percentage judged on sliced metrics, then ramp — with rollback to the previous version possible at every rung in seconds.

05Hundreds of models from many teams — a dedicated fleet each?

No: hot models get dedicated replicas; the long tail is bin-packed many-to-a-GPU with per-model quotas, so one team’s spike cannot starve another team’s model.

06What does a caller get when inference cannot finish in time?

A degraded answer inside the deadline — a cached prediction or a declared default — never a hang; the caller’s timeout is a contract the platform designs around.

01What latency budget does the hottest caller hand us?

The hot caller’s 50 ms p99 is decomposed into owned budgets — feature fetch ~10 ms, batch wait capped at 5 ms, inference ~10 ms — each stage with its own monitor.

02A canary turns bad at 2 a.m. — how fast is it gone?

A bad version is out of traffic in seconds: rollback is a registry pointer swap to the previous version, which stays loaded and warm until the new one has soaked at 100%.

03The registry or the feature store degrades — do predictions stop?

Serving survives control-plane trouble: replicas keep answering with the already-loaded version, and feature-store misses fall back to defaults with the degradation logged.

04What GPU utilization justifies the bill?

Batching and bin-packing hold GPU utilization around 40–60% — lower means paying for idle silicon, near-saturation means no headroom when traffic spikes.

05How do we notice a model quietly rotting?

Every model logs sampled inputs and predictions by default, so drift against the training snapshot is detectable before any ground-truth label arrives.