From sandbox to production: how to test whether an AI agent is actually contained

01

A successful demo is not a containment test

A sandbox is useful because it lets a team see an agent complete an intended task with safe data and cooperative tools. It is not, by itself, evidence that the agent will remain inside its intended role when the context becomes incomplete, contradictory, delayed, or operationally messy. The difference is not semantic. A demo asks whether the happy path works. AI agent containment testing asks what the system does when the happy path no longer authorises it to continue.

Suppose an internal operations agent can recommend a change after it reads telemetry. In a friendly sandbox, the data is current, the target is obvious, the tool is available, and a valid approval is present. A containment test deliberately changes one of those facts. The target belongs to a different environment. The approval has expired. The telemetry is stale. The tool returns a partial response. The handoff lacks a required identifier. The safe result may be a block, a request for evidence, or a human escalation—not a completed action.

The purpose is not to trick an agent into failure. It is to demonstrate that the product has a known boundary and a predictable response at that boundary. That is what makes a system more governable as its capabilities and integrations expand.

02

Begin with an authority map, not a prompt

Before you write a scenario, make an authority map for one concrete workflow. List what the agent may read, which decisions it may recommend, which tools it may call, which actions it may execute, what requires human approval, and which outcomes must always stop. Keep the map specific enough that an engineer and a risk owner would recognise the same boundary.

For a claims assistant, the map might allow reading the current claim record and drafting a recommendation. It might permit a tool call that retrieves policy terms. It might forbid changing payout details, disclosing a restricted field, or finalising a high-risk case without an approval record. This is already much better than a generic instruction to ‘be safe,’ because it supplies observable conditions for an AI agent evaluation framework.

Include the environment at this stage. The same capability may be permissible in a development fixture, constrained in staging, and subject to a stronger gate in production. If a test record has no product and environment attached, it becomes difficult to interpret later. A failure in a local experiment should not silently influence a production readiness decision.

• Purpose: the decision or assistance the agent is allowed to provide
• Inputs: approved data sources, classifications, tenants, and freshness requirements
• Actions: permitted tools, targets, parameters, budget, and expiry
• Approvals: decisions that require a named human or policy confirmation
• Safe stops: the conditions that must block or escalate rather than continue

03

Turn the map into one scenario with hard boundaries

Open Scenario Studio and describe the actual job, not the whole product. A useful first scenario has a problem statement, named roles, expected outputs, and a small number of parameters that can vary between cases. If the workflow has multiple agents, show the handoffs and the shared information explicitly. This makes it possible to test whether a required state disappears between the agent that creates it and the agent that acts on it.

Then use Constraint Engine to separate hard requirements from preferences. A hard constraint might say that a high-impact action requires a current approval identifier and must be blocked if it is absent. Another might require the target environment to match the approved environment. A soft constraint might prefer a shorter recovery path or lower tool cost. Keep the distinction honest: a risk boundary should not be reduced to a soft score because it is inconvenient to test.

Run feasibility and conflict checks before treating the scenario as a gate. A formal or structural check can show that the rules you have modelled are not obviously contradictory. It cannot prove that the agent will follow them. That proof comes from running representative cases, inspecting evidence, and repeating the test when the system changes.

const containedActionScenario = {
  product: "claims-triage",
  environment: "staging",
  hardConstraints: [
    "target_environment_matches_approval",
    "approval_is_current_before_execution",
    "restricted_fields_never_leave_authorized_scope",
    "missing_evidence_blocks_and_escalates"
  ],
  softConstraints: ["minimize_unnecessary_tool_calls"]
}

04

Test three cases for every important boundary

The first case is the ordinary pass. It has the required evidence, a permitted target, and an action inside scope. It confirms that containment does not make the product unusable. Capture the run as a baseline, including the scenario version, verifier version, environment, and evidence that made the action acceptable.

The second case is the near boundary. Something material is incomplete or uncertain, but not necessarily malicious: an approval is pending, a source is older than the accepted freshness window, a handoff contains a conflicting risk tier, or the right environment cannot be verified. The desired response is usually to request clarification, defer the action, or escalate. This case tests whether the agent recognizes uncertainty rather than improvising authority.

The third case is the safe-stop test. Remove a required field, offer an out-of-scope target, present an unauthorised tool path, or create an explicit policy conflict. The expected result must be recorded before you run it: block, quarantine, or escalate. A system that reaches a superficially useful answer by bypassing that condition has failed containment even if a general-purpose quality judge likes the prose.

05

Verify the path, not only the final answer

An agent can produce a careful final response after taking an unsafe path. It may retrieve an unpermitted record, call a tool against the wrong environment, or make an action attempt that later fails. Final-output evaluation alone misses that distinction. Where the workflow is consequential, capture the safe trace metadata you need to inspect the path: agent and model identifiers, tool names and outcomes, latency, error state, correlation ID, deployment identifier, and the references that connect the run to its product and scenario.

Do not use traces as an excuse to collect every prompt, output, or customer record. The right principle is minimum useful evidence. Preserve the identifiers, states, and event relationships that prove a boundary was respected or crossed; use redaction, metadata-only delivery, or hashing where full content is not needed. Privacy is part of containment, because a safety programme that creates a new sensitive-data exposure has moved the risk rather than reduced it.

Verifier Factory should make the path checks visible. A deterministic check can confirm that an approval field exists, that a tool target is allowed, or that a restricted action was not called. A semantic evaluator can assess whether an escalation explanation is understandable. Keep those roles separate, and report a missing trace or unavailable verifier as missing—not as an automatic pass.

06

Inject controlled faults before the environment does it for you

A contained system has to remain contained when dependencies are imperfect. Chaos Studio gives a team a controlled way to introduce a timeout, delay, authentication failure, missing tool response, stale state, or provider unavailability into a non-production experiment. The experiment should begin with a target, a narrow fault, a duration, an expected response, and an abort condition. Treat this as a safety exercise, not a performance stunt.

For each fault, ask what the agent should know and what it should not assume. If a policy lookup times out, can the system safely defer? If an approval service is unavailable, does it block rather than accept an old cache silently? If a fallback provider is used, does the same scope and evidence policy still apply? The answer may differ by product, but it must be explicit before the fault is injected.

When the experiment ends, inspect the run, the constraint results, the evidence state, the recovery action, and the alert. A failed dependency can be a passing containment outcome if the system stopped safely and created a useful record. Conversely, a technically recovered request can be a containment failure if it proceeded without the required proof.

07

Use a staged rollout to test production reality without treating users as test cases

No finite test suite can reproduce every production context. That is why containment testing continues after deployment through a staged rollout. Start with the narrowest permitted environment and authority. Monitor a small, approved sample. Compare its evidence with the pre-deployment baseline. Expand only when the controls, alerts, ownership, and response path have all worked under real operating conditions.

Monitoring needs more than a score threshold. It should know which product and environment it protects, which scenario or suite applies, what a critical violation means, who receives the alert, whether an RCA investigation should start, and how to distinguish a system failure from an evaluator or worker outage. A dashboard that shows a zero because data is unavailable creates false reassurance at exactly the moment a containment control needs attention.

A good rollout also creates a rollback and review decision before the first event occurs. Decide which boundary failures stop promotion, which open a Quality Issue, which require an owner to acknowledge them, and how a corrected run will be compared with the original. The result is not perfect assurance. It is a system that can reveal and limit its own uncertainty rather than hiding it.

08

The containment test is complete only when the learning persists

A single safe-stop test is a useful beginning, not a permanent property of the product. Every meaningful change can disturb the boundary: a model update, a new tool, a revised prompt, an altered handoff, a provider fallback, a permission change, or a new environment. Preserve the original failing or boundary case as a regression suite, then run it in CI or a controlled deployment gate whenever the relevant part of the system changes.

When a real incident reveals a new gap, use RCA to distinguish the observed facts from the proposed cause. Make the smallest repair you can verify. Add a scenario, constraint, contract check, or monitoring rule that would have caught the failure earlier. Record the evidence and ownership. This is how AI containment matures: not through a broad declaration that the system is safe, but through a growing body of boundary tests that continue to earn trust.

The simplest measure of progress is not how many safeguards the team can name. It is whether a colleague can take one important decision, remove the evidence or authority it depends on, and watch the system stop in the expected way—with a record clear enough to explain and improve. That is what production-ready AI containment looks like in practice.