How to Design Realistic Evaluation Scenarios for AI Agents

The easiest way to write a weak evaluation scenario is to start with a prompt: “Answer this customer,” “summarise this ticket,” “make a plan.” Prompts are visible. Work is not. The difficult part is everything around the prompt—the approval that may be missing, the record that has not been updated, the tool that returns an ambiguous result, the policy that quietly outranks the user’s request.

Start with a decision or action that someone already owns. “Decide whether this invoice can be paid,” “route a production incident,” and “recommend a patient appointment” are useful because they have consequences. They also give you people to interview. Ask what an experienced operator checks first, what they never assume, when they ask for help, and which mistakes are expensive enough to remember years later.

Those answers are the raw material of a scenario. They turn informal judgement into something a system can be evaluated against.

A realistic scenario is not a question with a little context attached. It is a small, deliberate model of the operating environment: the user’s request, the records available to the system, the tools it may call, the roles of the agents involved, the policies that govern them, the current state, and the shape of an acceptable result.

The word “unknown” deserves special attention. In production, an unknown value is often where risk begins. If the backup timestamp is unavailable, do not quietly replace it with a plausible timestamp. If the customer’s entitlement cannot be verified, do not let the agent infer one from tone or history. Mark the absence of evidence as part of the scenario and see whether the system notices.

This is also where time enters the picture. A scheduling decision at 9:00am may be safe at 4:00pm. A deployment approval may expire. A cached retrieval may be accurate yesterday and dangerously wrong today. Put the clock in the scenario when the clock changes the decision.

A rubric that says “be helpful and accurate” is almost impossible to defend. It rewards style, and style is not the same thing as reliability. A stronger rubric describes evidence: identify the incident, check the backup age, respect the change freeze, avoid the destructive tool without approval, and explain the safest next step when evidence is missing.

Notice that none of these assertions requires one exact paragraph. A reliable system may phrase its answer in several ways. What matters is whether it recognised the important facts and acted within the boundary.

Write the negative space too. Which tool must never be called? Which field must be redacted? Which claim must not be made? In high-consequence workflows, “what the agent refused to do” can be more informative than the prose it produced.

You do not need a thousand random variations. You need the variations that make the right action different. Remove the approval. Make the retrieval stale. Give the user less permission. Return two records with conflicting values. Let the tool time out after the agent has already formed a plan. Change the request halfway through.

These are not edge cases in the dismissive sense. They are often the moments when a workflow leaves its happy path and becomes a business event.

Pairwise thinking helps keep the suite useful. Instead of testing every possible combination of ten variables, identify the interactions with real consequences: urgency with approval, permission with tool type, memory with date, or customer tier with refund amount. A few well-chosen intersections reveal more than a warehouse of near-identical prompts.

A final answer is a poor witness. An agent can arrive at the correct conclusion through an unsafe route: querying a source it was not allowed to see, skipping a required check, or calling a destructive tool before asking for approval.

Capture the trajectory. Record the plan, the handoffs, the tool calls, the arguments sent to those tools, the state changes, and the final response. Then decide which steps are required, which are optional, and which orderings are unacceptable.

This is where multi-agent evaluation scenarios become especially valuable. A handoff is not just a message between two models. It is a boundary where context can be lost, permissions can be widened, and responsibility can become ambiguous. Test the handoff as carefully as you test the answer.

A scenario without an owner becomes an orphaned prompt. Add the business domain, failure mode, severity, and the person or team responsible for interpreting the result. Write down whether the scenario came from an incident, a near miss, a policy requirement, a customer complaint, or a design assumption.

When it fails, the failure should point somewhere. Is the prompt ambiguous? Is the orchestration wrong? Does a tool contract expose too much? Is the data stale? Is the evaluator measuring the wrong thing? The scenario does not need to answer every question, but it should make the next question easier to ask.

Also decide what failure means operationally. Some findings should be logged. Others should require approval, quarantine a release, or block deployment entirely. A score without a consequence is only decoration.

A scenario changes when the work changes. Version it like code. Pin the model, tool schemas, data snapshot, and rubric that produced a baseline. If a business rule changes, create a revision and explain why the new result should not be compared directly with the old one.

Review the suite after incidents and near misses. Add the uncomfortable case while the details are still fresh. Retire scenarios that no longer represent a real risk, but keep their historical results when they explain how the system got here.

The goal is not a giant library. It is a living set of tests that people trust enough to use before a release.

Before you publish an AI agent evaluation scenario, read it as if you were the person who has to act on the result. Is the decision real? Are the unknowns visible? Are permissions and tools clear? Can you tell what the agent was allowed to do? Would two reviewers agree on what counts as a serious failure?

Most importantly, can the scenario teach you something after the model changes? If the answer is no, it may still be a useful prompt for exploration, but it is not yet an engineering test.

The best scenarios feel less like exam questions and more like carefully reconstructed moments from real work. They leave room for a system to surprise you—but not room for you to misunderstand what happened.