Ad Ranking Evaluation Framework

01What must be written down at the moment an auction is decided?

Every impression logs the full auction context — candidates considered, position shown, model version, experiment assignment, and the propensity (the probability the serving policy gave that ad that slot) — because replay is worthless if we do not know why the old policy chose what it chose.

02Can we score a new model before it spends real money?

Yes: offline replay runs the candidate over held-out logged auctions and reports AUC (how often the model ranks a real click above a non-click), NDCG (whether the best items land at the top of the list), and a counterfactual revenue estimate with propensity correction — an offline gate every candidate passes before it touches live traffic.

03The model predicts 2% CTR — how do we know 2% means 2%?

Calibration reports bucket impressions by predicted CTR and compare against observed clicks, per segment; a model may not enter the auction until predicted and observed agree within tolerance, because bids multiply pCTR and miscalibration misprices every auction.

04How does an online test avoid corrupting advertiser budgets?

Budget-split design: each campaign’s budget is divided across arms in proportion to traffic, so test and control never drain the same wallet — with user-split and interleaving available where budget interference does not apply.

05CTR went up in the test — is that enough to launch?

No: every readout computes guardrails — revenue, advertiser ROI, user negative feedback, ad load, latency — and any breach blocks the launch regardless of how good the primary metric looks.

06A purchase lands 12 days after the click — does the experiment ever see it?

Yes: conversions are attributed within a configurable window (up to 28 days), late postbacks restate earlier readouts, and where privacy rules replace raw conversions with modeled estimates, the modeled-label error is tracked as a metric of its own.

01What stops us from declaring victory on noise?

Metrics are pre-registered with power math before traffic flows, and readouts use sequential correction — peeking daily at raw p-values manufactures false winners.

02Offline says +2%, online says flat — which pipeline is lying?

Metric definitions are versioned and shared by the offline and online pipelines, so an offline-online gap is attributable to bias, not to two teams computing revenue differently.

03How much advertiser money can one experiment put at risk?

Advertiser exposure is a capped, budgeted risk: traffic share limits it, budget-split accounting prevents double-spending, and a guardrail breach triggers rollback within minutes, not a meeting.

04Are fairness and policy checks advisory or binding?

Fairness and ad-policy checks (restricted categories, anti-discrimination rules) are launch blockers, and rollback authority sits outside the team shipping the model.

05The experiment ends — who is still watching?

Production stays watched after launch: calibration drift and prediction-distribution shift alert continuously, a ~1% persistent holdback measures the stacked long-term effect of every launch, and a small exploration slice keeps logged propensities non-zero so tomorrow’s offline evaluation stays possible.