LLM Evaluation Platform

01What goes into the golden set, and who decides?

The golden set — the versioned benchmark dataset every change is scored against — has four buckets, each earning its place: a stratified sample of real production traffic, an adversarial library, constructed edge cases, and replays of failures that actually shipped; the dataset is part of the system, versioned like code, with every addition reviewed like a code change.

02What triggers an eval run?

Every pull request touching a prompt, model version, or retrieval configuration triggers a full run against the pinned golden set in CI (continuous integration — the pipeline that checks every change before merge); no change class is exempt, because a one-line prompt edit regresses quality as silently as a model swap.

03Does every case burn judge tokens?

No — deterministic checks run first: format, schema validity, required citations, and refusal patterns are exact and free; the LLM judge (a model grading another model’s output against a rubric) scores only what rules cannot, and a case that fails a deterministic check never reaches the judge.

04How does the judge avoid grading on the wrong things?

Bias controls are built in, not optional: candidate order is randomized — or judged in both orders with agreement required — against position bias; scoring is length-controlled so verbosity cannot buy points; and a model is never judged by itself, because judges prefer their own prose.

05A run regresses — what exactly happens to the PR?

The gate blocks the merge when the aggregate score drops past threshold against the trend baseline — trends and aggregates, never single-case flips — and borderline results route to a human review queue instead of hard-blocking, with the reviewer’s verdict recorded against the run.

06Where do new cases come from after launch?

Production keeps the set alive: thumbs-down events, support escalations, and incident replays flow into a candidate queue, get clustered and deduplicated, and a human curator promotes the keepers into the golden set — the set grows from real failures, not synthetic imagination.

01Two runs of the same change disagree — which one is right?

Fully pinned and reproducible: an eval run is a tuple of versions — prompt, model, retrieval index snapshot, golden-set version, judge version, rubric — and re-running the tuple reproduces the score, so every number on the trend chart is re-derivable.

02How long may the gate hold a PR?

The gate fits a CI budget of about ten minutes: deterministic checks cost milliseconds, generation and judging fan out across cases in parallel, and the run completes in roughly a minute of wall clock — a gate engineers bypass is a gate the platform failed to design.

03What is the smallest regression we can honestly claim to catch?

Statistically honest: the platform publishes its minimum detectable regression per set size — a 200-case set cannot see a 3-point drop through ±6 points of comparison noise — and never blocks on a difference inside the noise band.

04How much can we trust the judge’s number?

Judge-human agreement, measured on a maintained human-labeled slice, is the published ceiling on every claim: at 90% agreement a true 5-point drop reads as 4, and any judge-model upgrade triggers recalibration plus a bridging run before its scores join the trend line.

05What does a run cost, and who notices when it drifts?

Metered and observable: deterministic-first keeps judge spend to the minority of cases that need it, cost per run is a dashboard number per team, and a run whose judge-token bill suddenly doubles is itself a regression signal.