01Does the agent answer everything, or triage first?
Triage first: every inbound message is classified for intent — answerable from knowledge, needs account data, or belongs with a human immediately (angry customer, legal threat, churn risk).
Free full guide
AI Customer Support Agent
Design an AI customer support agent for a SaaS product.
00
Practice checkpoints
The interview rhythm stays compact, so the page can spend attention on the actual design decisions.
- 01Clarify scope
- 02Requirements + scale
- 03API + data model
- 04Draw architecture
- 05Deep dive
- 06Trade-off decision
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02What may the agent actually DO to an account?
Only approved tools with typed inputs — check status, update settings, issue a bounded refund. Risky or irreversible actions require explicit confirmation or a human.
03When must a human take over, and what do they receive?
On low confidence, repeated failure, or a customer request, a human takes over. They get a handoff packet: the conversation, retrieved sources, attempted actions, and the agent’s own uncertainty. Never a cold start.
04Does the agent remember earlier turns - and earlier conversations?
Within a single conversation, it always remembers — even a very long thread. Across conversations, it keeps only account facts, never the earlier chat content, following the retention policy.
05One language or many?
The pipeline handles any language, but evaluation does not. Golden sets are built per locale, because answer quality does not carry over from one language to another.
06Can a customer demand a human at any point?
Always. 'talk to a human' is one turn away at every step. Hiding that exit inflates deflection metrics (the share of conversations that never reach a human) while destroying trust.
Out of scopeVoice channel and real-time speech · Training custom foundation models · Sales and marketing conversations (support only)
Non-functional requirements
01What is the worst failure mode — being slow, or being wrong?
Being wrong: an invented policy or an unauthorized refund costs more than any latency. Grounding, citation, and action gating outrank fluency.
02What data can the agent see while answering?
Only what this customer may see. Knowledge access is scoped per tenant and per plan. Internal runbooks and other tenants’ data must never surface in an answer.
03How fast should answers feel?
Answers must start appearing within about 2 seconds and finish well under 30 — and hold that at roughly 10K conversations a day.
04When something goes wrong, can we reconstruct why?
Yes: every turn is traced — query, retrieved chunks and scores, prompt, tool calls with inputs/outputs, and the escalation decision. Failures replay into the eval set.
05How do we know the agent is getting worse?
Quality is checked against a standing set of reference support conversations on every model or prompt change. Nothing ships unmeasured. Deflection rate alone is not quality.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- What is the escalation contract — how fast must a human respond after handoff?
- What are the data retention and PII rules for conversation logs?
- Is there a cost ceiling per conversation — at what point does the agent cost more than the human it replaces?
- Should we expect adversarial users — people deliberately trying to trick the agent into refunds or into leaking data?
- Are there seasonal or launch-day surges in support volume we must absorb?
02
Numbers that force architecture decisions
Treat every estimate as a pressure that justifies a component: cache, queue, partition, replica, worker pool, or fallback path.
Deflection economics
10K conversations/day (interviewer-supplied), agent fully resolves ~60%, a human ticket averages ~15 min10K × 60% = 6,000 resolved/day; 6,000 × ~15 min = 90,000 min ≈ 1,500 agent-hours/day; at 8-hour shifts ≈ 190 support seats absorbed
The business case lives or dies on resolution QUALITY — which is why the eval suite is a requirement, not tooling.
Context budget per turn
~8K tokens: system + policies (2K) + retrieved chunks (4K) + conversation window (2K)2K + 4K + 2K = 8K tokens per turn — retrieved chunks are half the entire budget, so those 4K tokens must carry the answer
Retrieval precision is the real quality lever — a bigger window is a cost increase, not a fix.
Latency decomposition
p95 targets: retrieval 300 ms + rerank 200 ms + first token 1.5 s≈ 2 s to first streamed token
Each stage gets its own budget and its own monitoring — "the AI is slow" is not a diagnosis.
Eval suite cost
500 golden conversations × 3 judged variants per release500 × 3 = 1,500 judged runs per release — run in parallel, that is minutes of wall-clock and a few dollars of judge-model tokens
Continuous evaluation costs less than one mishandled enterprise ticket.
Escalation load
~40% of 10K conversations escalate4,000/day routed to humans WITH handoff packets
Handoff quality determines whether humans trust the agent — a bad packet doubles their work.
Decision example
The numbers
Ten thousand conversations a day, and the agent fully resolves sixty percent. The entire value rests on those resolutions being RIGHT. One invented refund policy erases a month of savings.
My choice
Build the agent as a gated pipeline. It classifies intent first, retrieves with the customer’s own permissions, and answers only with citations. Every side effect goes behind typed, allow-listed tools, with risk tiers enforced outside the model. Confidence gates every step: below the threshold, the customer gets a human plus a handoff packet — the conversation, sources, attempted actions, and the agent’s uncertainty. Every turn is traced and replayable, and failures feed a golden eval set judged by rubric and pairwise comparison, calibrated by humans.
Avoid
What I would NOT do: hand the model a database connection and a system prompt that says "be helpful." Free-form access is how an agent refunds the wrong customer. I also would not measure quality by deflection rate alone. An agent that confidently deflects with wrong answers scores perfectly while destroying trust. And exact-match tests cannot judge support conversations. "Did it resolve correctly" is a preference judgment, which is why the eval set stores good AND bad reference outputs with rubrics.
Change if
If the product goes multi-region with data residency rules, retrieval indexes and conversation logs must shard per region. The eval suite then gains per-locale golden sets, because quality is not portable across languages.
03
Architecture path
One complete picture first, then each path as its own diagram — the write path and the read path carry different traffic and justify different components.
Complete picture
Overview — every component
Everything the model does passes a gate: retrieval is permission-scoped, actions go through the tool gateway, and low confidence exits to a human with a full handoff packet. Traces record every hop.
Path 1
Answer path — classify, retrieve, ground, respond
Intent classification runs before any generation. Retrieval carries the customer’s permissions into the index query. The answer cites its sources, or it does not ship.
Path 2
Action path — typed tools behind risk tiers
The model proposes; the gateway disposes. Low tiers execute, medium tiers ask the customer to confirm, high tiers require a human — and every call lands in the trace.
04
API and data model
Before optimizing, make the contract inspectable: endpoints, entities, ownership, retries, and state.
POST/support/messages
req{ conversation_id?, text }
res200 streamed answer with citations · or { escalated: true, ticket_id }
Customer identity comes from the session — the agent’s permissions are the customer’s permissions, never broader.
POST/support/{conversation_id}/escalate
res201 { ticket_id } with the handoff packet attached
Fires on low confidence, repeated failure, or explicit customer request — escalation is a feature, not a failure.
POSTinternal: tools.execute(tool, input, risk_tier)
resresult · blocked (needs confirmation) · denied (policy)
The only path from model output to real side effects: typed inputs, allow-listed tools, risk tiers enforced outside the model.
Core entities
Conversationconversation_id (PK) · customer_id · channel · state: active/escalated/resolved
Turnturn_id (PK) · conversation_id · role · content · trace_ref
trace_ref links to the full replayable trace: retrieval, prompt, tool calls, decisions.
ToolActionaction_id (PK) · conversation_id · tool · input · result · risk_tier · confirmed_by
Risky tiers record who confirmed — the agent, the customer, or a human agent.
HandoffPacketconversation_id · summary · retrieved_sources · attempted_actions · uncertainty_notes
What the human receives at escalation — the difference between a takeover and a restart.
05
Deep dive directions
Pick one lane for the final third of the interview. Each lane gives you the topic, the interviewer question it should answer, and the failure mode to avoid.
The agent promises a refund it should not
Ask
What are the layered defenses between "the model wants to refund $500" and money actually moving?
Answer
The gateway decides, not the model. The refund tool takes typed inputs with a hard cap, and risk tiers live in the tool gateway outside the model. $500 sits above the auto-execute line, so the money moves only after the customer confirms or a human approves. Even a blocked attempt lands in the trace.
Avoid
Trusting the system prompt as the control — policy lives in the tool gateway, outside the model.
Tenant data stays home
Ask
Two companies use this product. Walk retrieval for a question whose best-matching chunk belongs to the OTHER tenant.
Answer
Put the tenant ID inside the index query. Then the other tenant's chunk is never even a retrieval candidate — it's excluded before ranking, not pulled out afterward. Don't post-filter the results instead. Miss one check there and a competitor's document ends up in the prompt.
Avoid
Post-filtering retrieved chunks — the permission filter must be inside the index query itself.
The handoff moment
Ask
Confidence drops mid-conversation. What exactly does the human agent see, and what happens to the customer’s flow?
Answer
The human opens a handoff packet, not a blank screen. It carries the full conversation with a running summary up top, the sources the agent retrieved, every action it tried, and its own uncertainty note. The customer stays in the same chat and just sees a person join. They never repeat themselves.
Avoid
Escalating with just a transcript — sources, attempted actions, and uncertainty notes are what save the human from restarting.
Is it getting worse?
Ask
A prompt tweak ships. How do you know support quality did not silently regress — before customers tell you?
Answer
Replay a standing set of golden conversations on every release. A judge model scores each answer against a rubric and compares the new version against the old, and humans re-calibrate it regularly. Exact-match can't grade prose, and deflection rate rewards confident wrong answers. Rubric plus pairwise judgment catches the regression before customers do.
Avoid
Exact-match tests or deflection rate as the metric — support answers need rubric plus pairwise judgment, human-calibrated.
Who saw what
Ask
A customer disputes an action the agent took last month. Reconstruct the conversation: what is stored, for how long, and who may read it?
Answer
Every turn was traced — the query, retrieved chunks and scores, the prompt, tool calls with inputs and outputs, and the escalation decision. So the disputed action replays exactly as it happened. The logs live only for the retention window, with PII redacted. Reading them is itself a permissioned, audited action.
Avoid
Storing raw conversations forever with PII intact — retention and redaction are requirements, not cleanup.
Ready to practice?
Talk through AI Customer Support Agent out loud and get AI scoring on the explanation.