What an eval suite is, and how to build one

Eval suites are important because all too often someone makes a prompt/code change, improves one metric/benchmark, but unknowingly regresses something else.

A few saved prompts are not enough. If they only get run when someone remembers, they won’t catch the regression you didn’t think to check. An eval suite turns those examples into something repeatable: fixed inputs, fixed graders, and a run in CI.

Part 1 of the agent evals series covers the framework around observability primitives, what to evaluate at which level, and pass^k as a reliability metric. After that, you still have to decide which layers to start with, how to build the task set, how grading works, and how to get the suite onto your laptop and into CI.

I’d aim for around 50 well-specified tasks rather than hundreds of sloppy ones. Use deterministic graders if you can, make sure you calibrate an LLM judge against humans (if you choose to use one) and read through the failures yourself.

Table of contents

A suite, not a notebook

Saved examples are useful when you’re changing a prompt, but they’re not an eval suite by themselves. They become a suite when you can run the same inputs against the same graders in CI and get a clear answer about whether the change is safe to ship.

A suite has layers because different failures show up in different places. The first version I build usually has deterministic checks, a golden-task set, and production-trace replays. End-to-end agent evals come later, when the product has failures that only appear across turns or coordinated tool use.

Layer 0: read outputs yourself. Eyeball ten outputs in a notebook. This should be a habit you build: when you change a prompt, read a handful of outputs yourself and notice behaviour the graders haven’t been told to check for.

Layer 1: deterministic asserts. These are ordinary assertions on model output: validate the JSON schema, match a final-answer regex, check tool-call argument names, enforce “no PII in this field”, or require a single integer between 0 and 100. They give high precision on the narrow behaviour they cover and say nothing about the rest.

Before adding an LLM judge, look for the deterministic checks the feature already implies. Most non-trivial LLM features have several.

Layer 2: golden tasks. A frozen set of inputs paired with annotated good outputs or grading functions. Runs on every model or prompt change. If a deployment changes the model or prompt, I’d want it to pass this set first.

Each task should be narrow enough that you know what broke when it fails. I’d add some deliberately awkward cases too, like a user arguing with the policy, a request where the policy leaves room for judgment, or a case where the right answer is to refuse even though the request sounds reasonable.

Layer 3: production replays. Sample real user sessions from production logs, subject to your privacy policy. Replay them against the new model or prompt. Diff the output against the previous version. This is useful when current users no longer look like the examples in your golden set.

This layer isn’t trying to grade the new outputs, because you don’t have a gold answer for each replayed input. It surfaces cases where the new version behaves materially differently from the old one. Diffs that look weird get triaged into the golden set as new tasks.

Layer 4: agent eval. An end-to-end agent eval uses a simulated user and policy-aware grading across multiple turns. It’s slow and expensive, so I would keep it out of PR-time CI unless the product fails in ways that only show up across dialogue or coordinated tool use.

The layers don’t replace each other. Broken JSON should fail before you spend money running the golden set. When user behaviour shifts, production replays show the cases your frozen suite no longer covers. Dialogue and tool-coordination bugs need the slower multi-turn harness. If all you have is a golden set, you won’t know which of those classes you’re missing.

Building the dataset

The optimisers, dashboards, and LLM-as-judge calibrations don’t matter if the inputs are wrong. Most of the time, people pick an eval framework before they have a single golden task; I try to write five or ten tasks before I even compare tools.

For a fifty-task suite, I usually pull from four places.

Production traces. Start with logs you’re allowed to use and anonymise them before they enter the suite. I keep traces that cover different failure modes, especially when they check behaviour the agent needs to get right. I wouldn’t dump 200 random sessions into the suite and call it done; most of those cases will be routine paths you already handle.

Hand-written hard cases. You should also keep track of some bugs and edge cases that you’ve encountered. For example, if someone finds a bad agent response in Slack or during testing, turn that case into a task while the details are still fresh. Then revisit these cases when the product changes, because old failures can stop being useful.

LLM-synthesised cases. Synthetic cases are useful when you need breadth across categories or paraphrases, but I wouldn’t use them for the hard cases. Their biases match the synthesising model, and if your synthesiser is the same model you’re testing, you’ve created a self-evaluation problem. Use a different model family if you do this.

Adversarial / red-team. Try to break your own product. Half a day of trying to make the agent refuse a legitimate request, prompt-inject itself, leak the system prompt, or bypass its guardrails will yield more usable tasks than a week of automated synthesis. The cases you catch this way are usually the worst regressions you’d otherwise have shipped.

You shouldn’t build a suite from just one source. Production traces over-fit to current user behaviour; synthetic cases over-fit to whatever the synthesising model thinks failures look like. My current default for a new suite of fifty tasks is roughly thirty production-sourced cases, ten hand-written hard cases, five synthesised for breadth, and five adversarial.

Two sanity checks before a task enters the suite

First, write the reference solution. If you can’t write a good answer, sharpen the task before it enters the suite. If a colleague would grade the same input differently from you, neither of you can grade it consistently across releases either.

Second, decide what a failure looks like. Not just “wrong” (too coarse). Is this a “wrong tool” failure, a “wrong arguments” failure, a “right answer wrong format” failure, a “missed the question entirely” failure? Tasks whose failure modes you can’t enumerate aren’t ready yet.

Grading: how your LLM output gets scored

You can have good tasks and a working harness and still get a bad signal if the grader is wrong. The grader decides what the suite rewards. If it rewards the wrong thing, prompt iteration will push the model toward the wrong thing.

Start with deterministic checks. Use semantic judges only where exact checks don’t fit, and use humans to calibrate the judge. The rule of thumb is to use the weakest grader that can answer the question reliably.

Deterministic graders. Use these when the output has something you can check without interpretation: valid JSON, the right tool name, an argument in range, SQL that executes and returns the expected rows. They’re cheap, fast, and reproducible, so they should catch format and tool-call failures before you spend money on anything subjective.

Where they go wrong is pretending a semantic requirement is a string requirement. “Apologise when the policy says to apologise” is not the same as “include the word sorry”. The first one checks behaviour; the second one checks one wording. If the task is really about meaning, a regex will punish harmless paraphrases and teach the model to optimise for wording rather than behaviour.

Semantic graders (LLM-as-judge, embedding similarity). Use these when the output needs interpretation, such as a refusal that can be worded many ways or a summary that has to include certain facts without matching a reference answer exactly. An LLM judge is not a neutral measuring instrument. It’s part of the test, with its own prompt, model choice, biases, and cost.

Before I put an LLM judge in CI, I’d first check whether it agrees with humans. Pick a small batch of outputs and label them yourself. Then run the judge on the same batch and compare the labels. If the judge matches your labels on 90% of examples, it may be good enough to track changes. At 60%, it’s mostly noise. Cohen’s kappa and Krippendorff’s alpha are useful later, but I would start with raw agreement.

I also wouldn’t use the same model as both candidate and judge. Same-family judges tend to prefer outputs that look like their own writing. Use a different model family on the judge side, or run two judges from different families and look at the cases where they disagree.

Human grading. Humans are slow, so I wouldn’t make them grade the whole suite every time. I would use them to calibrate the judge and to spot drift. A useful review batch has some examples the judge passed, some it failed, and some where the judge disagreed with the expected label. The disagreements are usually the most useful part: they tell you whether the rubric is unclear, the judge is biased, or the task itself is under-specified.

For comparing two prompts or models, I would usually ask which output is better rather than asking for a 1-to-5 score. Pairwise judgments are easier for humans and LLM judges. Absolute scores are still useful if you want a long-running dashboard, but I would look at trends over several runs rather than treating one release as meaningful on its own.

Operations: when to run these evals

A suite that runs once a quarter is a suite that doesn’t catch regressions. It has to run somewhere, stay affordable, and keep old results comparable.

CI integration. Cheap layers (0, 1) run on every PR. Medium-cost layers (2, 3) run on every merge to main or pre-release. Expensive layers (4) run before major releases and model upgrades. If PR runs get expensive, people skip them. Keep the PR checks small enough that nobody has a reason to turn them off.

Cost management. A 200-task suite × five trials per task × $0.05 per call is $50 per CI run. Run that on every PR and you’re spending serious money on something nobody is reading. Cache LLM responses, either through the provider or in your own wrapper. For PRs, run a smaller slice of the suite, maybe 20 tasks. Save the full run for nightly or pre-release runs. Parallelise where you can, and use cheaper models in the grading harness when the candidate model is the only thing being tested.

Versioning. Inevitably, you’ll change the eval set. The important thing is knowing which version a historic run was scored on. Commit the eval set to git, tag it, and keep the task files close enough to the code that they get reviewed. If you change the suite, don’t compare the new score to an old run unless you’ve re-run the old model on the new tasks.

Two Failure Modes

Drift. Production behaviour shifts (a new user cohort, a new use case, a viral tweet about your product). Your eval distribution stops matching reality. The number on the dashboard stays good but production quality is degrading. Once a quarter, sample fifty new production traces, replay them against the current and previous model, diff the outputs, and manually triage anything weird. The triaged cases go into the golden set as new tasks.

Overfitting to the suite. Prompt iteration converges on “pass the suite” instead of “be good”. The model gets better at the specific tasks you scored and worse at the underlying capability. Keep a held-out validation set you don’t iterate on. The optimiser sees the trainset, you see the held-out set. When held-out scores stop tracking train scores, you’ve overfit. This is the same discipline as any ML training loop; people forget that prompt iteration is a training loop.

Tooling: use a framework if it helps, but make your own evals

You can use LangSmith, Braintrust, Arize Phoenix, Langfuse, Patronus, or a small runner you write yourself. Use a framework for storage, dashboards, run history, and integrations if it saves time. Don’t expect it to tell you which failures belong in the suite.

Keep the tasks, expected answers, and graders close to the product code. If a failure costs users, violates policy, or keeps showing up in testing, that knowledge should live in the eval set. A vendor can store the runs for you, but it can’t decide which cases belong there.

If you’re building this from scratch, start smaller than you think. Put each task in a JSON file. Register a few graders by name. Have a runner write scores to a CSV, then compare two run IDs to see what changed.

The version below is small on purpose: roughly 100 lines of Python across three files.

// tasks/geo_001.json
{
  "id": "geo_001",
  "input": "What is the capital of France?",
  "expected": "Paris",
  "grader": "exact_match"
}

# graders.py
GRADERS = {}

def grader(name):
    def deco(fn):
        GRADERS[name] = fn
        return fn
    return deco

@grader("exact_match")
def exact_match(output, task):
    return float(output.strip().lower() == task["expected"].strip().lower())

@grader("contains")
def contains(output, task):
    return float(task["expected"].lower() in output.lower())

# Add an llm_judge grader with the same signature when you need one.
# Calibrate it before wiring it into CI.

# run.py
import csv, json, glob, sys, uuid
from graders import GRADERS
from candidate import generate  # your model call: str -> str

def run(candidate_id, tasks_glob="tasks/*.json", out="runs.csv"):
    run_id = str(uuid.uuid4())[:8]
    with open(out, "a", newline="") as f:
        w = csv.writer(f)
        for path in glob.glob(tasks_glob):
            task = json.load(open(path))
            output = generate(task["input"])
            score = GRADERS[task["grader"]](output, task)
            w.writerow([run_id, task["id"], task["grader"], candidate_id, score])
    return run_id

def compare(run_a, run_b, path="runs.csv"):
    scores = {}
    for row in csv.reader(open(path)):
        rid, tid, _, _, score = row
        scores.setdefault(tid, {})[rid] = float(score)
    for tid, s in sorted(scores.items()):
        a, b = s.get(run_a), s.get(run_b)
        if a is not None and b is not None and a != b:
            print(f"{tid}: {a} -> {b}")

if __name__ == "__main__":
    cmd, *args = sys.argv[1:]
    if cmd == "run": print(run(args[0]))
    elif cmd == "compare": compare(args[0], args[1])

In the snippet, generate(input: str) -> str is your model wrapper: Anthropic, OpenAI, a local model, or whatever retry-and-cache layer you use. I left it out so the example doesn’t depend on one SDK. The compare function only prints diffs where both runs have a score. If a task errors in one run and completes in another, handle that separately rather than treating it as a score regression.

This is not production-grade, but it is enough to run the same tasks against two candidates and see what changed. From there you can add an LLM-judge grader, retries, parallelism, and a better report. A framework gives you storage, dashboards, history, and integrations once the task set is solid. It still won’t choose the failures for you.

What to do when an eval fails

Most of the time, nobody writes down what actually happens after CI fails. I think it’s useful to have a short investigation:

• Pull related failures.
• Read the traces.
• Confirm the cause.
• Change one thing, or a small set of related things.

For example, suppose a task starts failing in CI. Yesterday’s prompt/code passed, today’s fails, and the grader says: “wrong answer: expected Paris, got France.”

I’d pull a few similar failures and read the traces for that task, plus three or four others tagged with the same concept. If only one task failed, it may be noise. If the whole cluster moved, it’s probably a regression.

Then write down a hypothesis: maybe the prompt change weakened the “be specific” instruction, so the model is generalising answers. It’s only a guess, but it’s enough to test. Run the failing cluster against the previous prompt/code and the new one. If the cluster passes on the old version and fails on the new version, you know where to look. If both versions behave the same way, the change probably wasn’t the cause.

Only after that should you make the fix. Either revert the relevant change or modify the prompt to address the regression specifically. Run the failure cluster again, plus a small holdout, so you can see whether the fix addresses the regression without breaking nearby behaviour. Commit with a message that names the cluster and the behaviour you’re keeping.

Taking 5 to 15 minutes per failure is realistic if you do the investigation in that order. The common mistake is jumping from the failing task straight to a prompt edit, then making a fix that’s vibe-tested rather than an actual verified change.

Common anti-patterns in eval tests

1. Evals that are difficult to run. If the eval suite isn’t runnable as a CLI command, it will require someone to remember to run it manually before it actually gets run.

2. Hard examples that need human grading. If a task needs a person to grade it every single time, keep it in a manual-review pile rather than adding it to the automated eval set.

3. LLM-as-judge without calibration. You shouldn’t trust an off-the-shelf judge just because it worked for other evals. Check it against human labels for your own tasks before you plug it into your eval set.

4. The judge and candidate are the same model. Judges tend to prefer outputs that look like their own writing. Use a different model family for the judge, or compare two judges and look at where they disagree. Panickssery et al. is a useful paper on this self-preference problem.

5. Suite bloat. It is easy to add a task every time a regression slips through and much harder to remove old tasks later. Review the suite every so often and archive cases that no longer catch real regressions.

6. Optimising the suite, not the agent. If the dashboard number keeps improving but the agent feels worse when you try it yourself, the suite has probably stopped tracking what people actually care about. Refresh the tasks rather than tuning harder against the same set.

Conclusion

Initially, you won’t have everything. The first version just needs enough coverage to block a bad release and tell you if yesterday’s prompt change regressed any tasks. Add the slow evals when bugs are showing up in release testing that the CI checks can’t reproduce.

Once the suite is running, the follow-up posts in this series will cover:

• what comes next after the first suite is working
• calibrating LLM judges with Cohen’s kappa and the rest of the inter-rater stats
• how to iterate prompts without overfitting the suite

For the conceptual framework behind why this works, Part 1 of the agent evals series covers observability primitives, what to evaluate at which level, and pass^k as a reliability metric.

References

• Breaking Down Agent Evals (Part 1): A Practitioner’s Guide: the conceptual companion
• Prompts are Hyperparameters: what to do once you have a metric and a suite
• Yao et al. τ-bench. 2024. arXiv:2406.12045: the canonical end-to-end agent eval
• Anthropic. Demystifying Evals for AI Agents. anthropic.com/engineering/demystifying-evals-for-ai-agents