One Score Can Hide the Broken Product

5 minute educational book

One Score Can Hide the Broken Product

Turn a model benchmark into a map of user promises and failure slices

Created by Bob · AI-assisted and reviewed before publication

What you will learn

Translate a product promise into evaluation scenarios and metrics
Explain how aggregate scores can conceal important failures
Describe a versioned continual-evaluation loop

Page 1 of 3

Start With the Product Promise

A useful evaluation begins with behavior the application owes its users, not with a convenient public leaderboard. Convert that promise into scenarios: representative inputs, relevant context, allowed actions, expected properties, and important edge cases. A support assistant might need correct policy answers, calibrated refusal when evidence is absent, stable citation links, and acceptable latency.

Those are different requirements. HELM formalized a similar principle at model scale by evaluating multiple scenarios and metrics instead of treating accuracy as the whole story. For an application team, the key move is to write the contract before choosing the grader.

A single PRODUCT PROMISE fans into four scenario cards that each connect to a distinct metric dial; a detached LEADERBOARD trophy has no connection to the product path.

Page 2 of 3

The Average Conceals Slices

Suppose a system passes 92 percent of cases overall but fails most requests containing tables, recent policy changes, or a particular language. The average is mathematically valid and operationally misleading. Slice the evaluation set by properties tied to known risks: input type, task, source, length, language, user group, tool path, or adversarial pattern.

Each slice needs enough cases to support the conclusion; tiny samples should carry uncertainty instead of false precision. Compare both quality and non-quality metrics such as latency, cost, calibration, robustness, or safety when they matter. A release decision should reflect the weakest important promise, not only the best aggregate.

One large OVERALL score tile lifts to reveal four weighted SLICES beneath it; three are green while a TABLES slice is coral. Separate QUALITY and LATENCY meters prevent one blended score.

Page 3 of 3

Failures Become Permanent Tests

Production evaluation is a loop. Log or collect failures with appropriate privacy controls, remove sensitive data, reproduce the behavior, and turn recurring or costly failures into reviewed test cases. Version the dataset, prompts, model, retrieval corpus, tools, and grader so a change can be compared with a known baseline.

Automated graders make broad testing cheaper but can be biased or inconsistent, so calibrate them against human judgments and manually inspect consequential disagreements. When a fix improves one slice and harms another, keep both results. The durable asset isn't one launch score; it is a growing specification that makes regressions visible.

A RELEASE flows to observed FAILURES, then to REVIEWED CASES, then into a VERSIONED EVAL SET that gates the next release. HUMAN CALIBRATION checks an automated grader beside the loop.

Key takeaways

An application eval measures user promises instead of leaderboard prestige
Risk-relevant slices reveal failures hidden by averages
Reviewed production failures should become regression tests

Check your understanding

Why can a high overall pass rate still describe a broken product?: Important input slices or user promises may fail even when larger or easier groups dominate the average.
Why calibrate automated graders against people?: Automated graders can be biased or inconsistent, especially on consequential or ambiguous cases.

Sources

These references were used to check the important factual claims in this edition.