How to build an eval suite

When building agents, it is imperative you have a way to measure whether your agent actually works/improving against your previous iteration. Evals are what you run your agent against to measure how well your agent is doing. An eval suite is a set of tools that helps you run your agents against those evals.

Eval suites are important because all too often someone makes a prompt/code change, improves one metric/benchmark, but unknowingly regresses something else. A few saved prompts aren’t usually enough. If you only choose to run some of them when you remember to, you won’t catch the regression you didn’t run. An eval suite turns your evals into a repeatable process through fixed inputs, fixed graders, and a run in CI.

Part 1 of the agent evals series covered the framework around observability primitives. After that, you still have to build the eval set, decide how grading works, and how to get the suite onto your laptop and into CI. For v1 I’d aim to run around 50 well-specified tasks rather than hundreds of lower-quality ones. Use deterministic graders if you can, make sure you calibrate an LLM judge against humans (if you choose to use one) and read through the failures yourself.

Table of contents

A suite, not a notebook

Having a few eval cases to run is useful when you’re making a small prompt change. It gives you a quick sanity check, but that isn’t an eval suite. For that, you want to know that you can run the same inputs repeatedly as part of deployment and have a metric that tell you whether the change is improving your agent.

One thing I liked from Demystifying Evals For AI Agents is that evals need multiple layers. No single type of layer is going to catch all the potential issues.

To do this I often start pretty simple. I first add deterministic checks for things I can measure directly (e.g. exact matches, ranges, etc.). I then add an LLM judge for cases that are harder to score with fixed rules (e.g. extracting facts from paragraphs). After that, I sample traces from production and replay those. Once I have a good idea of where the agent is doing well and where it isn’t, I start adding more expensive end-to-end evals, like multi-turn evals or tests that require coordinated tool use.

Layer 0: Manual transcript debugging. This should be the first step in every evaluation cycle. Unfortunately, most people skip this part and go straight to running end-to-end evals. In this layer, when you make a change to a prompt or any other part of your agent, read through some of the outputs yourself and try to find behaviour the graders haven’t been told to look at.

Layer 1: Deterministic asserts. This layer involves simple assertions on the model output. For example:

• Matching a final-answer regex (e.g., the name Picasso is matched or is contained in the answer to the question “Who painted Guernica”)
• Checking tool-call argument names to ensure certain tools are called
• Enforcing no PII is found in the output
• Arithmetic questions

This layer gives you high precision for certain checks and can also miss things like misspellings and minor deviations, but is much simpler to implement and runs much more quickly.

Layer 2: Golden tasks. This layer involves a set of inputs paired with an annotated set of good outputs or graders. This should run on every prompt or model change. Each task should be narrow enough for you to be able to figure out what broke.

It’s also a good idea to add some ambiguous cases (e.g., a user getting angry at the model, or an unspecified query by the user) to test how the model responds. This makes the eval cases reflect actual usage patterns a little better.

Layer 3: Production replays. At this stage, we sample production user traffic (in accordance with your company’s privacy policy). We can then run these queries against the prompt/model change and compare the output.

This is particularly useful when your production traffic distribution no longer looks like the examples in your golden task set.

This layer isn’t about trying to grade new outputs because there isn’t a gold answer for each input. However, it does surface cases where the change you’ve made behaves materially differently from the old one. This is also an opportunity to generate more test cases to feed back into your golden task set.

Layer 4: Agent evals. Finally, we can use agent evals to simulate a user and a grader to evaluate the model across multiple turns. This layer is slow and expensive, so I wouldn’t keep this in the PR runtime CI unless you’ve seen the product fail in ways that only show up in multi-turn or multi-tool use cases.

Overall, the different layers are meant to complement each other and provide a strong foundation for your eval set. For example:

• Broken JSON and malformed answers should fail in Layer 1 before you run your golden set in Layer 2.
• Layer 3 helps you when your user behavior shifts, since production replays will surface examples your frozen set in Layer 2 no longer covers.
• For multi-turn, multi-tool coordination, you need the slower multi-turn agent eval harness.
• If you just have one layer, you won’t be able to identify which of these is happening.

Building the dataset

The optimisers, dashboards, and LLM-as-judge calibrations don’t matter if the inputs are wrong. Most of the time, people pick an eval framework before they have a single golden task; I try to write five or ten tasks before I even compare tools.

For a fifty-task suite, I usually pull from four places.

Production traces. Start with logs you’re allowed to use and anonymise them before they enter the suite. I keep traces that cover different failure modes, especially when they check behaviour the agent needs to get right. I wouldn’t dump 200 random sessions into the suite and call it done; most of those cases will be routine paths you already handle.

Hand-written hard cases. You should also keep track of some bugs and edge cases that you’ve encountered. For example, if someone finds a bad agent response in Slack or during testing, turn that case into a task while the details are still fresh. Then revisit these cases when the product changes, because old failures can stop being useful.

LLM-synthesised cases. Synthetic cases are useful when you need breadth across categories or paraphrases, but I wouldn’t use them for the hard cases. Their biases match the synthesising model, and if your synthesiser is the same model you’re testing, you’ve created a self-evaluation problem. Use a different model family if you do this.

Adversarial / red-team. Try to break your own product. Half a day of trying to make the agent refuse a legitimate request, prompt-inject itself, leak the system prompt, or bypass its guardrails will yield more usable tasks than a week of automated synthesis. The cases you catch this way are usually the worst regressions you’d otherwise have shipped.

You shouldn’t build a suite from just one source. Production traces over-fit to current user behaviour; synthetic cases over-fit to whatever the synthesising model thinks failures look like. My current default for a new suite of fifty tasks is roughly thirty production-sourced cases, ten hand-written hard cases, five synthesised for breadth, and five adversarial.

Two sanity checks before a task enters the suite

First, write the reference solution. If you can’t write a good answer, sharpen the task before it enters the suite. If a colleague would grade the same input differently from you, neither of you can grade it consistently across releases either.

Second, decide what a failure looks like. Not just “wrong” (too coarse). Is this a “wrong tool” failure, a “wrong arguments” failure, a “right answer wrong format” failure, a “missed the question entirely” failure? Tasks whose failure modes you can’t enumerate aren’t ready yet.

Grading: how your LLM output gets scored

You can have good tasks and a working harness and still get a bad signal if the grader is wrong. The grader decides what the suite rewards. If it rewards the wrong thing, prompt iteration will push the model toward the wrong thing.

Start with deterministic checks. Use semantic judges only where exact checks don’t fit, and use humans to calibrate the judge. The rule of thumb is to use the weakest grader that can answer the question reliably.

Deterministic graders. Use these when the output has something you can check without interpretation: valid JSON, the right tool name, an argument in range, SQL that executes and returns the expected rows. They’re cheap, fast, and reproducible, so they should catch format and tool-call failures before you spend money on anything subjective.

Where they go wrong is pretending a semantic requirement is a string requirement. “Apologise when the policy says to apologise” is not the same as “include the word sorry”. The first one checks behaviour; the second one checks one wording. If the task is really about meaning, a regex will punish harmless paraphrases and teach the model to optimise for wording rather than behaviour.

Semantic graders (LLM-as-judge, embedding similarity). Use these when the output needs interpretation, such as a refusal that can be worded many ways or a summary that has to include certain facts without matching a reference answer exactly. An LLM judge is not a neutral measuring instrument. It’s part of the test, with its own prompt, model choice, biases, and cost.

Before I put an LLM judge in CI, I’d first check whether it agrees with humans. Pick a small batch of outputs and label them yourself. Then run the judge on the same batch and compare the labels. If the judge matches your labels on 90% of examples, it may be good enough to track changes. At 60%, it’s mostly noise. Cohen’s kappa and Krippendorff’s alpha are useful later, but I would start with raw agreement.

I also wouldn’t use the same model as both candidate and judge. Same-family judges tend to prefer outputs that look like their own writing. Use a different model family on the judge side, or run two judges from different families and look at the cases where they disagree.

Human grading. Humans are slow, so I wouldn’t make them grade the whole suite every time. I would use them to calibrate the judge and to spot drift. A useful review batch has some examples the judge passed, some it failed, and some where the judge disagreed with the expected label. The disagreements are usually the most useful part: they tell you whether the rubric is unclear, the judge is biased, or the task itself is under-specified.

For comparing two prompts or models, I would usually ask which output is better rather than asking for a 1-to-5 score. Pairwise judgments are easier for humans and LLM judges. Absolute scores are still useful if you want a long-running dashboard, but I would look at trends over several runs rather than treating one release as meaningful on its own.

Operations: when to run these evals

A suite that runs once a quarter is a suite that doesn’t catch regressions. It has to run somewhere, stay affordable, and keep old results comparable.

CI integration. Cheap layers (0, 1) run on every PR. Medium-cost layers (2, 3) run on every merge to main or pre-release. Expensive layers (4) run before major releases and model upgrades. If PR runs get expensive, people skip them. Keep the PR checks small enough that nobody has a reason to turn them off.

Cost management. A 200-task suite × five trials per task × $0.05 per call is $50 per CI run. Run that on every PR and you’re spending serious money on something nobody is reading. Cache LLM responses, either through the provider or in your own wrapper. For PRs, run a smaller slice of the suite, maybe 20 tasks. Save the full run for nightly or pre-release runs. Parallelise where you can, and use cheaper models in the grading harness when the candidate model is the only thing being tested.

Versioning. Inevitably, you’ll change the eval set. The important thing is knowing which version a historic run was scored on. Commit the eval set to git, tag it, and keep the task files close enough to the code that they get reviewed. If you change the suite, don’t compare the new score to an old run unless you’ve re-run the old model on the new tasks.

Two Failure Modes

Drift. Production behaviour shifts (a new user cohort, a new use case, a viral tweet about your product). Your eval distribution stops matching reality. The number on the dashboard stays good but production quality is degrading. Once a quarter, sample fifty new production traces, replay them against the current and previous model, diff the outputs, and manually triage anything weird. The triaged cases go into the golden set as new tasks.

Overfitting to the suite. Prompt iteration converges on “pass the suite” instead of “be good”. The model gets better at the specific tasks you scored and worse at the underlying capability. Keep a held-out validation set you don’t iterate on. The optimiser sees the trainset, you see the held-out set. When held-out scores stop tracking train scores, you’ve overfit. This is the same discipline as any ML training loop; people forget that prompt iteration is a training loop.

Tooling: use a framework if it helps, but make your own evals

You can use LangSmith, Braintrust, Arize Phoenix, Langfuse, Patronus, or a small runner you write yourself. Use a framework for storage, dashboards, run history, and integrations if it saves time. Don’t expect it to tell you which failures belong in the suite.

Keep the tasks, expected answers, and graders close to the product code. If a failure costs users, violates policy, or keeps showing up in testing, that knowledge should live in the eval set. A vendor can store the runs for you, but it can’t decide which cases belong there.

If you’re building this from scratch, start smaller than you think. Put each task in a JSON file. Register a few graders by name. Have a runner write scores to a CSV, then compare two run IDs to see what changed.

The version below is small on purpose: roughly 100 lines of Python across three files.

// tasks/geo_001.json
{
  "id": "geo_001",
  "input": "What is the capital of France?",
  "expected": "Paris",
  "grader": "exact_match"
}

# graders.py
GRADERS = {}

def grader(name):
    def deco(fn):
        GRADERS[name] = fn
        return fn
    return deco

@grader("exact_match")
def exact_match(output, task):
    return float(output.strip().lower() == task["expected"].strip().lower())

@grader("contains")
def contains(output, task):
    return float(task["expected"].lower() in output.lower())

# Add an llm_judge grader with the same signature when you need one.
# Calibrate it before wiring it into CI.

# run.py
import csv, json, glob, sys, uuid
from graders import GRADERS
from candidate import generate  # your model call: str -> str

def run(candidate_id, tasks_glob="tasks/*.json", out="runs.csv"):
    run_id = str(uuid.uuid4())[:8]
    with open(out, "a", newline="") as f:
        w = csv.writer(f)
        for path in glob.glob(tasks_glob):
            task = json.load(open(path))
            output = generate(task["input"])
            score = GRADERS[task["grader"]](output, task)
            w.writerow([run_id, task["id"], task["grader"], candidate_id, score])
    return run_id

def compare(run_a, run_b, path="runs.csv"):
    scores = {}
    for row in csv.reader(open(path)):
        rid, tid, _, _, score = row
        scores.setdefault(tid, {})[rid] = float(score)
    for tid, s in sorted(scores.items()):
        a, b = s.get(run_a), s.get(run_b)
        if a is not None and b is not None and a != b:
            print(f"{tid}: {a} -> {b}")

if __name__ == "__main__":
    cmd, *args = sys.argv[1:]
    if cmd == "run": print(run(args[0]))
    elif cmd == "compare": compare(args[0], args[1])

In the snippet, generate(input: str) -> str is your model wrapper: Anthropic, OpenAI, a local model, or whatever retry-and-cache layer you use. I left it out so the example doesn’t depend on one SDK. The compare function only prints diffs where both runs have a score. If a task errors in one run and completes in another, handle that separately rather than treating it as a score regression.

This is not production-grade, but it is enough to run the same tasks against two candidates and see what changed. From there you can add an LLM-judge grader, retries, parallelism, and a better report. A framework gives you storage, dashboards, history, and integrations once the task set is solid. It still won’t choose the failures for you.

What to do when an eval fails

Most of the time, nobody writes down what actually happens after CI fails. I think it’s useful to have a short investigation:

• Pull related failures.
• Read the traces.
• Confirm the cause.
• Change one thing, or a small set of related things.

For example, suppose a task starts failing in CI. Yesterday’s prompt/code passed, today’s fails, and the grader says: “wrong answer: expected Paris, got France.”

I’d pull a few similar failures and read the traces for that task, plus three or four others tagged with the same concept. If only one task failed, it may be noise. If the whole cluster moved, it’s probably a regression.

Then write down a hypothesis: maybe the prompt change weakened the “be specific” instruction, so the model is generalising answers. It’s only a guess, but it’s enough to test. Run the failing cluster against the previous prompt/code and the new one. If the cluster passes on the old version and fails on the new version, you know where to look. If both versions behave the same way, the change probably wasn’t the cause.

Only after that should you make the fix. Either revert the relevant change or modify the prompt to address the regression specifically. Run the failure cluster again, plus a small holdout, so you can see whether the fix addresses the regression without breaking nearby behaviour. Commit with a message that names the cluster and the behaviour you’re keeping.

Taking 5 to 15 minutes per failure is realistic if you do the investigation in that order. The common mistake is jumping from the failing task straight to a prompt edit, then making a fix that’s vibe-tested rather than an actual verified change.

Common anti-patterns in eval tests

1. Evals that are difficult to run. If the eval suite isn’t runnable as a CLI command, it will require someone to remember to run it manually before it actually gets run.

2. Hard examples that need human grading. If a task needs a person to grade it every single time, keep it in a manual-review pile rather than adding it to the automated eval set.

3. LLM-as-judge without calibration. You shouldn’t trust an off-the-shelf judge just because it worked for other evals. Check it against human labels for your own tasks before you plug it into your eval set.

4. The judge and candidate are the same model. Judges tend to prefer outputs that look like their own writing. Use a different model family for the judge, or compare two judges and look at where they disagree. Panickssery et al. is a useful paper on this self-preference problem.

5. Suite bloat. It is easy to add a task every time a regression slips through and much harder to remove old tasks later. Review the suite every so often and archive cases that no longer catch real regressions.

6. Optimising the suite, not the agent. If the dashboard number keeps improving but the agent feels worse when you try it yourself, the suite has probably stopped tracking what people actually care about. Refresh the tasks rather than tuning harder against the same set.

Conclusion

Initially, you won’t have everything. The first version just needs enough coverage to block a bad release and tell you if yesterday’s prompt change regressed any tasks. Add the slow evals when bugs are showing up in release testing that the CI checks can’t reproduce.

Once the suite is running, the follow-up posts in this series will cover:

• what comes next after the first suite is working
• calibrating LLM judges with Cohen’s kappa and the rest of the inter-rater stats
• how to iterate prompts without overfitting the suite

For the conceptual framework behind why this works, Part 1 of the agent evals series covers observability primitives, what to evaluate at which level, and pass^k as a reliability metric.

References

• Breaking Down Agent Evals (Part 1): A Practitioner’s Guide: the conceptual companion
• Prompts are Hyperparameters: what to do once you have a metric and a suite
• Yao et al. τ-bench. 2024. arXiv:2406.12045: the canonical end-to-end agent eval
• Anthropic. Demystifying Evals for AI Agents. anthropic.com/engineering/demystifying-evals-for-ai-agents