Free full guide
Design an LLM inference gateway that sits between internal product teams and multiple model providers/backends, handling routing, streaming, quotas, and safety controls at company scale.
The interview rhythm stays compact, so the page can spend attention on the actual design decisions.
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
01Who calls this — every product team, or a few?
Every team calls it. The gateway is the only door to the model providers. Each team gets one API key and each use case gets one policy, so failover, quotas, and redaction live in one place instead of forty.
02A provider starts failing — whose problem is it?
The gateway owns it. Each use case declares a fallback chain of schema-compatible models, health-checked per provider. Responses carry model_used and fallback_reason, so callers can tell which model answered.
03Do users watch the answer appear word by word — does time to the first word matter to the product?
Yes — tokens pass through the gateway the moment the provider emits them, never buffered; the gateway meters tokens mid-stream so quota and cost see streaming traffic too.
04Are limits counted in requests or in tokens?
Tokens. A request can cost 100 tokens or 100,000, so each team gets a tokens-per-minute rate and a monthly dollar budget. Counting requests would miss the real cost.
05The same prompt arrives a thousand times — generate a thousand times?
No — an identical repeat may be answered once and reused, but only within the team that asked; personalized or user-specific traffic must always be generated fresh.
06The provider ships a new model version — who upgrades, and when?
Per-team policy. A team can pin a version for stable output, or take upgrades as the provider ships them. Pinned teams get a countdown the day the provider deprecates their model.
Out of scopeTraining or fine-tuning models (serving traffic only) · Building the GPU serving stack itself (that is the provider’s side) · End-user identity and sessions — callers are internal services
01How much latency may the gateway itself add?
The org runs about 2M requests a day, and the gateway adds only ~10-20 ms on the request path. First-token latency is the product metric. The gateway must never be the reason it doubles.
02How fast must a dying provider be detected?
A dying provider is detected within seconds, per provider and per model. How many seconds is a product decision. Every second of delay is more user requests failing.
03What gets logged from prompts and responses?
Metadata is always logged: tokens, model, cost, latency, trace id. Prompt and response content is logged only after the PII (personally identifiable information) redaction hook runs. Raw prompts in logs would turn the gateway into a compliance incident.
04How hard is a team's budget cap — if requests arrive in a burst right at the limit, is a small overshoot tolerable?
The cap is absolute. A hundred parallel requests must not each spend the last dollar. A team's budget can never be overshot by a burst, because these are real invoiced dollars.
05Spend spikes at 2 a.m. — how fast do we know who and why?
Every call lands in a usage ledger: team, use case, model, tokens, cost, and cache and fallback flags. Attribution is queryable in near real time. You do not wait to reconstruct it from provider invoices at month end.
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
Treat every estimate as a pressure that justifies a component: cache, queue, partition, replica, worker pool, or fallback path.
Cost attribution — tokens, not requests
One team: 200,000 requests/day, ~3,000 input + 800 output tokens each, at $3 per 1M input and $15 per 1M output tokens600M input × $3 + 160M output × $15 = $1,800 + $2,400 = $4,200/day ≈ $126K/month
A per-request rate limit cannot see this bill at all. Quotas and budgets must be counted in tokens, with input and output priced separately.
Failover detection budget
Org volume is ~2M requests/day (given in the NFRs): 2M ÷ 86,400 s ≈ 23 requests/s average, and a workday peak of ~4× average ≈ 100 requests/s, mostly on the primary provider; the breaker uses a 10 s sliding window with a 20% error threshold100 requests/s × 10 s ≈ 1,000 calls in the window; in a total outage, 20% of 1,000 = 200 errors accumulate after 200 ÷ 100 ≈ 2 s — so ≈ 200 requests fail before the breaker opens
The window length sets how much user pain you accept. A shorter window trips faster but flaps on one-second blips. Pick it as a product number, and pair it with half-open probes.
Streaming vs buffered first token
An 800-token answer generated at ~60 tokens/s; a buffering gateway holds the full response before forwardingbuffered first byte: 800 ÷ 60 ≈ 13 s; passthrough first token: ~0.5-1 s → roughly 15-25× worse perceived latency
Every synchronous hop added to the stream path is paid on all requests — the stream must be inspected in motion, never parked.
Semantic cache economics
Org-wide 2M requests/day, 15% combined exact + semantic hit rate, ~$0.021 provider cost per request (from the token math above)300,000 hits × $0.021 ≈ $6,300/day ≈ $190K/month saved; a hit answers in ~50 ms instead of ~13 s
Real money and a huge latency win — but every hit is a chance to serve the wrong answer, so keys must include tenant, model version, and template version.
Queue depth in a brownout
A brownout halves the primary provider’s throughput for 5 minutes at the ~100 requests/s peak: 100 arriving, 50 served, nothing shedbacklog grows (100 - 50) × 300 s = 15,000 requests; at ~20 requests/s spare capacity after recovery, draining takes another ~12 minutes
Queueing everything turns 5 minutes of brownout into ~18 minutes of degradation and 15K open connections — shed batch traffic and fail interactive traffic over instead.
Decision example
Two million requests a day across forty teams, at roughly two cents each, is about $42,000 a day of provider spend. And each of those providers browns out for a few minutes some week of the month.
I would build one gateway path. It authenticates the team, reserves max_tokens against its budget, checks the exact-match then semantic cache, routes by the use case's model chain with a per-provider error-rate breaker, and streams provider tokens straight through while metering them mid-stream. Usage settles into a per-team ledger. Prompt content touches logs only after the redaction hook runs. Model versions are per-team policy — pinned with a deprecation countdown, or auto-upgraded behind an eval canary.
What I would NOT do. First, let teams call providers directly — that is forty implementations of failover and zero shared attribution. Second, count requests for quota, since a 100-token ping and a 100,000-token agent loop are not the same spend. Third, buffer streams to inspect them. A gateway that waits for the full response turns a 1-second first token into 13 seconds of silence.
If there is one team, one model, and four-figure monthly spend, a thin shared client library with retries is the honest answer. The gateway earns its complexity only when teams, providers, or spend multiply.
One complete picture first, then each path as its own diagram — the write path and the read path carry different traffic and justify different components.
Complete picture
The hot path is auth → budget reserve → cache → route → stream; ledger settlement and health bookkeeping sit off it. A breaker trip changes only the router’s choice — callers keep the same contract and simply see model_used change.
Path 1
The gateway counts tokens as they flow and settles the ledger from reserved to actual at stream end — quota and cost see streaming traffic exactly like blocking calls.
Path 2
The fallback must be schema-compatible with the primary, and the response says which model answered — silent substitution breaks every team that depends on output shape.
Before optimizing, make the contract inspectable: endpoints, entities, ownership, retries, and state.
POST/v1/responses
req{ use_case, messages, max_tokens, stream: true, output_schema? }
resSSE token stream; trailers carry model_used, token counts, cost, cache/fallback flags
Team identity comes from the API key, never the request body — one contract in front of every provider, so switching providers never touches product code.
GET/v1/usage (team_id, window=month)
res{ tokens, cost_usd, by_model, by_use_case }
Cost attribution as a product feature — team leads and finance read this, not provider invoices.
PUTinternal: policy(team_id, changes)
reshot-reloaded policy: pins, budgets, model chain, kill switch
Turning off a runaway team is a config write, not an emergency deploy.
Core entities
TeamPolicyteam_id (PK) · use_case · model_chain (primary → fallbacks) · version_policy: pinned/auto · tokens_per_min · monthly_budget_usd · redaction_profile
Hot-reloadable — a budget change or a kill switch must not wait for a deploy.
UsageLedgerEntryrequest_id (PK) · team_id · model_used · input_tokens · output_tokens · cost_usd · cache_hit · fallback_reason?
The attribution source of truth — written as a reservation pre-call, settled to actuals post-call.
ProviderHealthprovider + model · error_rate (sliding window) · p95 latency · breaker: closed/open/half-open
What the router reads before every dispatch; half-open probes decide when traffic is allowed back.
CacheEntryembedding key · tenant_scope · model + template version · response · ttl
A hit that ignores any one of these keys is how a stale or cross-tenant answer ships.
Pick one lane for the final third of the interview. Each lane gives you the topic, the interviewer question it should answer, and the failure mode to avoid.
A stream dies at token 400 of 800 with a provider 500. What does the client see, and can you fail over an in-flight generation?
End the stream with an explicit error event. Never go silent, and never splice in tokens from another model — it won't finish the same half-written sentence. You can't resume a generation mid-flight, so failover is an explicit restart on the fallback. Settle the dead call in the ledger for just the 400 tokens it produced.
Silently regenerating on provider B and splicing tokens — the client already rendered half an answer, and the new model will not produce the same continuation; resume is a restart and must be explicit.
One team’s runaway agent loop burned the org’s monthly budget in two days. Walk the controls that should have existed, in the order they should have fired.
Reserve-then-settle checks fire first. Each call reserves its max_tokens up front and settles to real use after, so parallel loops can't all spend the same last dollar. Next, a per-team tokens-per-minute cap slows the loop. A burn-rate alert in dollars per hour catches it on day one — the monthly cap is just the backstop.
A monthly invoice as the only control — per-call token metering, per-team caps, and alerts on burn RATE (dollars per hour against baseline) catch this on day one, not day thirty.
A user changes one critical word in the prompt and still gets the cached answer to the OLD prompt. Where did the semantic cache go wrong, and what bounds it?
The mistake was trusting similarity alone. Negation flips meaning at 0.98 cosine similarity, so a near-identical prompt is not the same prompt. Match exact first and let the semantic tier miss when unsure. Scope every entry by key: tenant, model version, and template version. Keep personalized traffic out, and give each hit a TTL.
Similarity threshold as the whole safety story — negation flips meaning at 0.98 cosine similarity; keys need tenant, model, and template version, personalized traffic stays out, and hits carry a TTL.
The provider announces your pinned model dies in 90 days. Whose problem is that, and what does the gateway’s upgrade path look like?
The gateway surfaces it; the team decides. The countdown starts the day the provider announces, not at day 90. Then an eval gate: replay the team's golden prompts on the new version, canary a small live slice, cut over on their sign-off. Silently auto-upgrading a pinned team kills the stability they pinned for.
Auto-upgrading everyone silently — teams pinned for output stability get an eval gate on their own golden prompts, a canary slice, and a sign-off, with the countdown surfaced from day one.
Safety adds a synchronous moderation call before generation and first-token latency doubles. How do you keep the guardrail AND the latency budget?
Run the checks in parallel, not in front. Fire input moderation alongside routing and the provider dispatch, and cancel generation if it fails. Its latency then hides behind work already happening. For output, scan the stream in rolling windows with a hard mid-stream cut-off — don't buffer a 13-second answer to inspect it at the end.
Serializing every check in front of the provider call — input checks run in parallel with routing, and output moderation scans the stream in rolling windows with a cut-off, instead of buffering the whole answer.
Talk through LLM Inference Gateway out loud and get AI scoring on the explanation.