Technical guide · counterfactual AI diagnosis
Counterfactual diagnosis: finding the smallest fix for an AI failure
When an agent fails, changing five things at once makes the next result impossible to interpret. Counterfactual diagnosis asks a sharper question: what is the smallest change that would have prevented this failure?
This tutorial uses plain language first and introduces technical terms only when they help. Read it with a small example from your own AI work in mind—a support agent, planner, researcher, or other ai agent.
01
What a counterfactual changes
A counterfactual keeps the original scenario and execution context as stable as possible, then changes one meaningful variable. That variable might be a prompt instruction, a model, a tool response, a retrieved document, a handoff field, or a constraint.
The purpose is not to create an imaginary perfect run. It is to test causality. If changing the handoff field makes the downstream agent complete the task, that is stronger evidence than changing the model, prompt, and tool response together.
02
Choose the variable that the evidence points to
Begin with the root-cause candidates from the trace. If the first unexpected event is a missing field, test the field. If the tool returned an error and the agent continued as if it succeeded, test the error-handling branch. If the plan violated a hard rule before any tool call, test the constraint or planner instruction.
Do not start with the most expensive change. A model swap can be useful, but it should not be the default explanation for every failure in an AI system.
- Prompt or system instruction
- Model or model version
- Tool output or availability
- Memory or retrieved context
- Agent-to-agent contract
- Constraint or verifier rule
03
Write a controlled experiment
Give the baseline and counterfactual the same scenario ID, input, and evaluation configuration. Name the changed variable and record its old and new value. If the experiment uses a simulated tool response, label it clearly so nobody mistakes it for production evidence.
const experiment = {
baselineRun: "run_original",
change: {
variable: "planner.handoff.requiredFields",
before: ["decision"],
after: ["decision", "reason", "nextStep"]
},
keepConstant: ["scenario", "model", "tools"]
}04
Compare more than pass or fail
A counterfactual that passes the original verifier may still be a poor repair. Compare feasibility, completeness, optimality, critical violations, tool calls, latency, cost, and trace shape. A prompt that forces a long answer might improve completeness while creating verbosity or cost problems.
The smallest fix is the one that removes the target failure while preserving the system’s other important properties. That is why a scorecard is more useful than a single green badge.
05
Promote a verified fix into a regression check
Once the counterfactual is convincing, turn it into a test. Keep the original failure as a regression fixture, add the new expected property to the verifier or contract, and run the case in CI. Monitor the production version for recurrence.
If the change affects a multi agent interface, update the contract and rerun the surrounding scenarios. If the failure came from a tool dependency, consider a Chaos Studio experiment to make sure the recovery path is robust rather than merely lucky.
06
A beginner’s experiment matrix
Create a tiny table with one baseline and two candidate changes. The baseline is the original failing run. Candidate A changes the handoff schema. Candidate B changes the final prompt. Keep the scenario, model, tools, and evaluator constant. Run both candidates and compare not just pass or fail, but the exact failure and the side effects.
If Candidate A fixes the missing field and leaves the rest stable while Candidate B only makes the response longer, the evidence favours the schema change. You have learned something about causality without needing to redesign the whole AI system.
| Run | One change | Result | Side effect |
|---|---|---|---|
| Baseline | none | missing field | original failure |
| A | require handoff field | passes | none observed |
| B | add prompt reminder | partial | longer response |07
When a counterfactual is not enough
Sometimes two changes interact, or the failure is caused by random tool availability. In that case, one successful counterfactual is not proof. Repeat the experiment, widen the regression set, and test the recovery path under controlled faults.
The goal is not to produce a clever explanation. It is to earn enough evidence to choose a repair confidently, then encode that learning in a verifier, contract, scenario, or monitoring rule so the same question does not have to be answered from scratch next week.
Where to go next
Keep the loop small: make one change, rerun the evidence, and only then widen the system.