There is a large gap between an agent demo and an agent in production. The demo takes an afternoon. Production takes several weeks, and almost all of the difference is in work that has nothing to do with prompting.
This is the process we follow, written out in full, so you can either run it yourself or know what to expect from whoever does.
Stage 1: Audit the real process
Not the documented process. The real one.
Sit with the people who do the work and watch them do it. Write down each step, which system it happens in, what information moves, and how long it takes. Then ask the question that produces the most valuable answers: "what do you do when it's not straightforward?"
The exceptions are where agent projects succeed or fail. Every process has them, almost none are written down, and an agent that handles the happy path but falls over on the twenty percent of awkward cases will not be trusted.
Output of this stage: a written process map with timings, systems, and a catalogue of exceptions.
Stage 2: Scope ruthlessly
Now decide what the agent owns. The strong recommendation: one workflow, completely.
The temptation is always to build something broad — an agent that handles "customer enquiries". Resist it. An agent that completes appointment rescheduling end to end, reliably, every time, is worth far more than one that partially handles six categories and needs checking on all of them. Partial automation often costs more than none, because someone now has to verify the output.
For each step in the process map, decide:
- Agent owns it — fully automated
- Agent drafts, human approves — good for anything customer-facing early on
- Human owns it — judgement, relationships, high stakes
Then write the escalation rules: under what conditions does the agent stop and hand over? Be specific. "When unsure" is not a rule. "When the customer mentions a complaint, a refund over £200, or legal action" is.
Stage 3: Connect the tools
An agent is only as capable as the tools it can call. Typical tool set for a business agent:
- Read from the CRM, database, calendar, inbox
- Write to the same systems, with scoped permissions
- Search over internal documentation
- Notify a human via email or Slack
Two engineering notes that matter more than they sound.
Give each tool a narrow, well-described interface. book_appointment(patient_id, slot_id) is a good tool. run_sql(query) is a loaded gun. The narrower the tool, the smaller the blast radius when the model makes a poor choice.
Use scoped credentials. The agent should authenticate as itself with the minimum permissions required, never as an administrator. When something goes wrong you want the damage bounded by the credential, not by the prompt.
Stage 4: Guardrails and escalation
This is the stage that separates a production agent from a demo.
Permitted action list. Explicit, enumerated, enforced in code rather than requested in the prompt. A prompt saying "never issue refunds over £500" is a suggestion. A check in the refund tool that rejects amounts over £500 is a guarantee.
Confidence thresholds. Where the agent is uncertain, it escalates rather than guesses. Models do not naturally express uncertainty well, so this usually means structured checks — did the lookup return exactly one match? Does the request fall inside known categories?
Rate and volume limits. An agent in a loop can do a lot of damage quickly. Cap actions per run and per hour.
Human fallback with context. When it escalates, a person should receive the conversation, what the agent attempted, and why it stopped — not just a flag.
Stage 5: Build the evaluation set
Do this before you build the agent, not after. It feels like a detour and it is the highest-leverage hour in the project.
Collect thirty to a hundred real historical cases with known correct outcomes. For a support agent, that is past tickets and how they were resolved. For a scheduling agent, past booking requests and the bookings that resulted.
Now you can answer the question that otherwise has no answer: did that change make it better? Without an evaluation set, every prompt adjustment is a guess, and improving one behaviour while silently breaking another is the normal outcome.
Run the suite on every change. Treat a regression the way you would treat a failing test.
Stage 6: Shadow run
Deploy the agent so it receives real inputs and decides what it would do — but does not act. Log the decisions. Have the team compare them to what they actually did.
This is the step teams skip under time pressure and the one that most reliably prevents an embarrassing launch. Shadow running surfaces the cases nobody thought of, with zero risk, against genuine traffic rather than your imagination of it.
Run it until the disagreement rate is low enough that you are comfortable, and until the disagreements that remain are ones you can explain.
Stage 7: Launch narrow, widen slowly
Do not switch everything over at once.
Start with a slice: one appointment type, one enquiry category, one customer segment, or simply ten percent of volume. Watch the logs daily for the first week. Read actual transcripts rather than dashboards — the summary statistics will look fine long before the behaviour is right.
Widen when the logs justify it. In practice most agents reach full volume within three to six weeks of launch, with a handful of prompt and tool adjustments along the way.
What this costs in effort
Rough proportions on a typical project, which surprise people:
- Process audit and scoping — 25%
- Integration work — 30%
- Guardrails, escalation, permissions — 15%
- Evaluation and testing — 20%
- Prompting and model work — 10%
The part everyone imagines is the whole project is a tenth of it. That ratio is the single most useful thing to understand before commissioning one.
The failure modes to watch for
- Scope creep during build. Every stakeholder wants one more case handled. Each one reduces reliability on the core path.
- No evaluation set. You will be unable to distinguish improvement from regression.
- Over-permissioned tools. Convenient during development, dangerous in production.
- Launching without shadow running. The cases you did not think of will find you.
- Treating it as finished at launch. Agents need tuning as real behaviour diverges from expectation.
If you have a process in mind and want a view on whether it is a good candidate — including an honest answer if it is not — describe it to us.
