Without evaluation, every change to an AI agent is a guess. Someone adjusts a prompt, the output looks better on the two examples they tried, and nobody knows whether three other behaviours quietly broke.
This is the difference between an agent you can improve and one you can only fiddle with.
Build the set first
Before writing the agent, collect real cases with known correct outcomes.
Where they come from: historical support tickets and their resolutions, past booking requests and the bookings that resulted, documents and the data that was extracted from them, calls and what the receptionist actually did.
How many: thirty is enough to be useful. A hundred is comfortable. More than a few hundred becomes slow to run and expensive, and the marginal value drops.
Doing this before the build feels like a detour and is the highest-leverage hour in the project. It forces you to define what correct means — which is a question that frequently reveals the team does not agree.
What goes in the set
Not just the easy cases. A useful suite is deliberately uncomfortable.
The common path, in proportion to real volume. If seventy percent of enquiries are one type, the set should reflect that.
Edge cases that actually occurred. The awkward ones from your real history.
Cases where the right answer is "I don't know." Questions your content cannot answer, requests outside scope. Critically important — see below.
Cases that should escalate. A complaint, an emergency, a request above a threshold.
Ambiguous inputs. Misspellings, incomplete information, two people with the same name.
Adversarial inputs. Text attempting to redirect the agent's behaviour.
Score task completion, not similarity
A common mistake is comparing the agent's output to a reference answer and scoring the textual overlap. This measures the wrong thing — a correct answer phrased differently scores badly, and a wrong answer phrased similarly scores well.
Score what actually matters:
- Did it take the correct action? Was the right appointment booked, the right record updated, the right value extracted?
- Was the information correct? Facts checkable against source.
- Did it escalate when it should have?
- Did it avoid doing anything it should not?
For agents, the action taken is the output. Judge that.
Make refusal a correct answer
The detail that most determines whether your agent is trustworthy.
If your scoring rewards answering and penalises "I don't know", you will optimise directly toward confident invention. Models are trained to be helpful; a refusal feels unhelpful; your metric will push it to answer anyway.
Include cases where declining or escalating is correct, and mark a confident answer to those as a failure. An agent that says "I don't have that information, let me connect you with someone" is doing exactly its job.
Run it on every change
Treat the suite as a test suite:
- Run before any prompt change ships
- Run when you change model or model version
- Run when you add or modify a tool
- Run when the knowledge base is re-indexed
- Run on a schedule in production, since upstream models change underneath you
A regression blocks the change. Not "we'll look at it later" — the same discipline you would apply to a failing unit test.
Without this, improving one behaviour while silently breaking another is the normal outcome, and you discover it from a customer.
Metrics that mean something
Task completion rate. The headline number: proportion of cases completed correctly end to end.
Appropriate escalation rate. Of the cases that should have escalated, how many did?
False confidence rate. Of the cases where the correct answer was a refusal, how many got a confident answer instead? This is the most important safety metric and the one usually missing.
Cost per case. Tokens consumed. A change improving accuracy by two percent while tripling cost is a trade, not a win.
Latency. Particularly for voice, where it determines whether the thing is usable at all.
Metrics to distrust
Average output quality rated by a model. Useful as a rough signal, unreliable as a gate. Models rating models share biases, and the score drifts when either changes.
Volume handled. Activity is not value.
Automation rate alone. An agent automating ninety percent while erring in five is worse than one automating sixty cleanly.
A single aggregate accuracy figure. It hides which category is failing, which is the thing you need to know.
Using a model as a judge
For open-ended outputs where there is no single correct string, having a model score the response against criteria is a reasonable technique with real caveats.
It works better when: the criteria are specific and checkable, the judge sees the source material, and you have spot-checked its scoring against human judgement.
It works worse when: the criteria are subjective, the judge is the same model that produced the output, or nobody has validated that the judge agrees with humans.
Treat it as a cheap approximation that lets you run frequently, with periodic human review to confirm it has not drifted.
Shadow running
Before launch, run the agent against real traffic without letting it act. Log what it would have done. Compare to what the team actually did.
This surfaces the cases nobody thought of, against genuine behaviour rather than your imagination of it, at zero risk. It is the step teams skip under time pressure and the one that most reliably prevents an embarrassing launch.
Run it until disagreements are rare and the ones remaining are explicable.
Production monitoring
Your evaluation set is built from the past. Real users produce inputs you did not anticipate.
In production:
- Log every interaction with inputs, actions and reasoning
- Sample and review weekly rather than waiting for complaints
- Track escalation rate over time — a sudden change means something shifted
- Make it easy to flag a bad outcome, and read the flags
- Feed real failures back into the evaluation set, so the same mistake cannot recur
That last point is what makes the suite improve rather than stagnate. Every production failure becomes a permanent test.
What good looks like
A mature agent setup has:
- An evaluation set of real cases, including refusals and escalations
- Automated scoring on task completion
- The suite run on every change, blocking regressions
- Production logging with weekly human review
- Failures fed back into the suite
That is a day or two of work to establish and it is the difference between an agent you can improve confidently and one that degrades quietly over successive edits.
If you have an agent in production without this, get in touch — it is usually a short engagement with a large effect on reliability.
