What an AI agent needs before production

An agent books a meeting in a demo. Before it books meetings for customers, it also needs to handle a full calendar, a revoked account, an ambiguous date, and a booking request that times out after the meeting was created.

That is the release decision: can the system complete the task, recognize when it cannot, and preserve enough progress for someone to finish or undo the work? A successful demonstration answers only part of that question.

We would start with one clearly defined workflow and expand its authority only after the failure paths work. Anthropic's guide to building agents makes a useful distinction: use a predictable workflow when the steps are known, and give a model control when the task needs that flexibility.

TL;DR Start with one workflow. Check permissions before actions, verify that the work happened, and make failures recoverable before expanding access.

Define what completion means

Write the outcome in terms the business can verify. For a booking assistant, a fluent confirmation is insufficient. The correct event must exist in the correct calendar, with the intended attendees and time.

Record which system proves completion, which actions need approval, and which outcomes require a person. An agent that prepares a refund for review has a different risk profile from one that sends money.

Avoid judging everything with a single success score. Track completed work, incorrect actions, unresolved requests, and unnecessary escalations separately. Otherwise an agent can appear safer simply by refusing useful work.

Put permission checks before actions

The model can propose an action. Application code must decide whether the authenticated user may perform it, on that account, with those arguments.

Check ownership and limits at the tool boundary. Do not rely on a prompt telling the agent to stay within policy. Retrieved documents, emails, and tool responses can contain instructions that should never gain authority over the application.

Each tool should define the inputs it accepts, how it reports failure, and what result confirms that the action happened. A successful HTTP response with an empty result should not become a customer-facing claim that work finished.

Make interrupted work recoverable

Failures between steps are different from failures inside a model response. If a payment provider accepted a request but the connection dropped, retrying blindly can duplicate the payment.

Persist progress before and after consequential actions. Where available, use an idempotency key, a request ID that lets the provider recognize retries of the same action. Check uncertain outcomes against the provider's records before trying again. Put limits on retries, runtime, and spending. After the limit, create a visible exception with enough context for someone to resolve it.

Failure	Safe next step
Read fails temporarily	Retry within a bounded policy
Write times out	Check whether it already happened
Required information is missing	Ask for it or send the case to review
Permission check fails	Deny the action in code
Repeated attempts make no progress	Stop and preserve the work for recovery

Restarting the agent is not automatically recovery. It must know what already happened.

Test realistic requests and costly exceptions

Build an evaluation set from representative requests, difficult cases, and past failures. Include incomplete information, conflicting instructions, tool outages, and attempts to access another customer's data.

OpenAI's evaluation guidance recommends task-specific evaluations and ongoing evaluation as the system changes. Use code to check facts such as whether the result contains the expected record ID. Use human review for judgments that require context, and validate automated graders against those reviews.

Longer context does not remove the need for testing. Chroma's context-rot research reports performance changes as input grows across its evaluated tasks. Test your own history and retrieval strategy rather than treating the model's maximum window as a quality guarantee.

Release with a way back

Begin with a restricted set of users and actions. Keep a record of each run, including model decisions, tool results, approvals, and the final business outcome. Keep secrets and unnecessary personal data out of logs.

Measure latency and cost per completed task, including failed attempts and review work. A cheap successful call can belong to an expensive failed workflow.

Before increasing traffic, demonstrate that an operator can disable actions, recover pending cases, and switch to a tested fallback. The fallback might be a manual queue rather than another model. It needs an owner and enough capacity to handle the diverted work.

Start the release review with one actual request: follow it from arrival to the calendar, database, or other service that records the result, then interrupt it at each consequential step. That exercise produces a concrete list of what needs fixing before launch.

Sources

• Anthropic: Building effective agents
• OpenAI: Evaluation best practices
• Chroma: Context rot