AI evals that actually catch regressions
Most teams building with large language models have an evaluation suite. Most of those suites are theater. They run a handful of hand-picked examples, assert exact string matches, and produce a green dashboard that means little about real product quality. When the model changes, the prompt shifts, or the retrieval pipeline is updated, the suite does not catch the regression. Users find it first.
This post is about building evaluations that actually work: evaluations grounded in real user behavior, run often enough to block bad changes, and structured so that failures become actionable engineering work rather than vague signals.
The cost of eval theater
Evaluation suites fail in predictable ways. The first failure mode is exact-match assertions. Language model outputs are not deterministic, and even when they are stable, there are many correct ways to express the same answer. Asserting that the output equals a golden string fails on harmless paraphrases. Teams respond by loosening the assertion until it accepts almost anything, or by freezing prompts and models to keep the string stable. Neither response improves the product.
The second failure mode is a tiny dataset. Twenty examples is not an evaluation. It is a demo. With a small sample you cannot distinguish a real regression from random variation, and you cannot see whether a fix for one case broke another category of requests. Small datasets also encourage overfitting: prompts get tuned to pass the examples rather than to serve users.
The third failure mode is missing failure analysis. A report that says "87% pass" tells you almost nothing. You need to know which examples failed, what kind of failure occurred, whether the failures cluster around a feature or input type, and how the current run compares to a baseline. Without traces and categorization, an evaluation score is a vanity metric.
The fourth failure mode is stale data. Evaluation sets written once and never updated stop reflecting the queries real users send. Products evolve, new features appear, and user language drifts. A suite that only tests yesterday's behavior will miss tomorrow's regressions.
A real-world lesson: Lucy the real estate assistant
Hamel Husain describes a case study from Rechat, a software-as-a-service product for real estate professionals. Rechat built Lucy, a conversational assistant intended to let users manage tasks such as contracts, listing search, creative assets, and appointments without navigating the full interface. Early progress came quickly through prompt engineering, but performance plateaued as the assistant's surface area grew.
The symptoms were familiar. Fixing one failure mode caused others to appear. The team had limited visibility into effectiveness across tasks beyond informal spot checks. Prompts grew long and unwieldy as they tried to cover edge cases and examples. The breakthrough came from treating evaluation as the center of the improvement loop rather than an afterthought.
This pattern matters because it shows what evaluation is for. Evaluation is not a certification step at the end of development. It is the mechanism that makes iteration reliable. When quality measurement, debugging, and system changes reinforce each other, the team can move quickly without guessing. When evaluation is weak, teams spend time on prompt tweaks and model swaps without knowing whether the product is improving.
Level 1: Unit tests and assertions
The first level of evaluation is fast, cheap assertions that run during development. These look like software unit tests, but they need to be designed for language model behavior. The goal is not to demand identical text. The goal is to assert properties that must hold for the feature to be correct.
Scoped tests work best when you break a capability into features and scenarios. In the Lucy example, a listing finder feature converts a user request into a query against a customer relationship management system. A useful test suite covers scenarios such as exactly one matching listing, multiple matching listings, and no matching listings. The assertions verify outcomes that matter: the number of results, the presence of required fields, or the absence of data that must not be shown.
Some assertions are generic rather than feature-specific. Rechat used tests to ensure internal identifiers were not exposed in user-facing output. A language model might be instructed not to include a universally unique identifier, but an assertion can verify that the response does not contain that pattern. Generic assertions like format checks, safety constraints, and required structure are valuable because they catch broad regressions across many features.
Writing these tests requires realistic inputs. Synthetic test generation is useful here. A model can generate plausible user instructions, including paired cases such as creating a contact and then retrieving it. The generated cases should be challenging and representative of how users interact with the system, not just easy happy-path examples. Over time, production traces and observed failures should feed back into the test set.
A practical detail is that language model unit tests do not always need a 100% pass rate. The acceptable pass rate is a product decision that depends on the task, the cost of failure, and the maturity of the system. What matters is that the tests run often, results are tracked over time, and regressions are visible before they reach users.
Level 2: Human and model evaluation
Assertions cannot cover everything. Open-ended generation, summarization, planning, and conversational assistance often require judgment about usefulness, completeness, tone, and factual grounding. That is where the second level of evaluation comes in: structured review of traces by humans and, when appropriate, by judge models.
The prerequisite is trace logging. A trace captures the sequence of events in an interaction: user messages, model responses, tool calls, retrieved context, and intermediate decisions. Without traces, debugging becomes guesswork. With traces, you can inspect what the system saw, what it produced, and where the behavior diverged from expectations.
Looking at traces has to be low friction. Hamel emphasizes removing obstacles to reading data, including building domain-specific views when generic tooling is not enough. For Lucy, useful context included the feature under test, the scenario, whether the input was synthetic or from a real user, filters across tools and scenarios, and links to related systems. The lesson is not that every team needs the same dashboard. The lesson is that evaluators need enough context on one screen to make a sound judgment quickly.
Human evaluation should start simple. Binary labels such as good or bad are often easier to apply consistently than complex scoring schemes. As the team learns more, labels can become more granular, but early complexity can slow review and hide disagreement. The key practice is to read real traces regularly, especially around failures, new features, and prompt or model changes.
For open-ended tasks, a judge model can scale review. The judge is a second model call that scores or critiques a candidate output against a rubric. The rubric must be specific. A vague question like "Is this a good answer?" invites inconsistent judgments. A better rubric checks concrete requirements: whether required facts are present, whether the response follows constraints, whether it avoids unsupported claims, or whether it uses the right format.
Judge models have biases. They may prefer longer answers, be influenced by formatting, or apply criteria inconsistently. Calibration against human ratings on a held-out set helps detect those biases. When the judge model, rubric, or task changes, calibration should be revisited. Raw agreement with humans can be misleading, especially when failures are rare; precision and recall for the failure class often give a clearer picture of whether the judge is useful.
A lightweight workflow can make this manageable. Hamel describes using a spreadsheet to compare a model response, a critique model's assessment, and a human reviewer's judgment. The human can record whether the critique was right, what the desired response should be, and why. This creates a feedback loop for improving the evaluator prompt or process. The evaluator is itself a system that needs evaluation.
Level 3: A/B testing and production feedback
The third level is A/B testing in production. Unit tests and trace reviews measure whether the system behaves as intended, but product decisions also depend on user outcomes. An A/B test can show whether a change improves task completion, reduces retries, increases adoption, or lowers support burden. These experiments are slower and more expensive than lower-level evaluations, so they usually come after the team has confidence in basic correctness and safety.
Production feedback should also flow back into earlier levels. Real user traces reveal edge cases that synthetic data misses. Support tickets and user complaints can be turned into regression tests. When a failure appears in production, the team should be able to find the trace, classify the root cause, add an assertion or rubric case if possible, and verify that the fix prevents recurrence. Mature evaluation systems make debugging faster because the observability and test infrastructure already exist.
Dataset construction and maintenance
A good evaluation dataset is curated, not accidental. The strongest examples often come from production logs, sampled and labeled to reflect actual usage. Production-derived data includes the messy inputs that designers did not anticipate: ambiguous requests, missing context, unusual formatting, and combinations of constraints.
When production data is limited, synthetic data can fill gaps. The useful pattern is to generate variations from seed scenarios, then filter and review them. Synthetic examples are not a substitute for reality, but they are better than leaving important scenarios untested. The filtering step matters: generated cases should be checked against assertions, deduplicated, and reviewed so the dataset does not become a pile of plausible-looking noise.
Metadata makes datasets more valuable. Each example can carry a category, difficulty level, feature area, source, and expected behavior. With metadata, teams can run targeted suites for a feature, compare performance across categories, and see whether a regression is narrow or systemic. Dataset maintenance should be continuous. Every significant failure is a candidate for a new test case.
Running evals in CI
Evaluations should run as part of the development workflow, not as an occasional manual exercise. A prompt change, retrieval adjustment, model upgrade, or tool schema edit can alter behavior in subtle ways. Running evaluations in continuous integration makes those changes visible close to the moment they are introduced.
Not every evaluation needs to run on every pull request. Fast unit-style assertions can gate routine changes. A broader model-graded suite can run on a schedule or before major releases. The split depends on latency, cost, and risk. The important principle is regression detection against a baseline. A drop beyond an agreed threshold should trigger investigation, and in high-risk paths it should block the change.
Tracking results over time turns evaluation into a diagnostic tool. A dashboard or stored history can show accuracy, failure counts, and category-level trends across versions of prompts, models, and data. When a metric moves, the team can ask whether the product changed, the dataset changed, or the evaluator changed. Without history, every run is interpreted in isolation.
Fine-tuning and debugging as byproducts
A strong evaluation system creates useful byproducts. One is better fine-tuning data. The same traces, labels, critiques, and filtered synthetic examples used for evaluation can become training data when the team needs to teach a model a style, format, or rule that prompting cannot enforce reliably. Fine-tuning works best when it is grounded in carefully curated examples rather than broad, unreviewed corpora.
Another byproduct is faster debugging. When a user reports a bad answer, the team needs to locate the trace, inspect retrieved context and tool calls, identify the failure category, and reproduce the issue. Evaluation infrastructure supports that workflow: searchable traces, assertions that flag common errors, domain-specific viewers, and regression tests that confirm the fix. The overlap between evaluation and debugging is large because both depend on observability and clear quality criteria.
Conclusion
Evaluations are the difference between engineering and guessing. A useful suite does not prove that a model is intelligent in the abstract. It shows whether a specific product behavior is reliable enough for the job it is supposed to do. The path to that reliability is incremental: fast assertions for obvious constraints, human and model review for nuanced behavior, production experiments for user outcomes, and a feedback loop that turns failures into better tests.
The hardest part is deciding what "correct" means for your task. Writing that down in tests, rubrics, and datasets forces the team to make quality explicit. It is uncomfortable work, but it pays off every time a regression is caught before users see it.
Sources
- Your AI Product Needs Evals by Hamel Husain (hamel.dev)