Root Cause Analysis for AI Agents: Moving Beyond "It Hallucinated"

A final-output check can show that a response violated policy, omitted a required action, or selected the wrong plan. It cannot show whether the planner misunderstood the objective, a specialist received incomplete context, a tool returned stale data, or a verifier approved an invalid handoff.

Those causes require different fixes. Prompt changes help when instructions are ambiguous. Schema validation helps when a handoff is malformed. Retrieval changes help when authoritative context is absent. A stronger model helps only when the model capability is the actual limiting factor.

Good diagnosis starts with an evidence inventory: agent messages, tool requests and responses, timestamps, constraint results, state mutations, and the final decision. The system then aligns these artifacts into one ordered trace. This prevents the loudest downstream symptom from being mistaken for the root cause.

Separate four levels of explanation. The symptom is what a user or evaluator observed. The fault is the classified system behavior. The mechanism explains how that behavior changed state. The root cause is the earliest controllable condition that made the mechanism possible. "The quote was too high" is a symptom. "Constraint blindness" is a fault. "The pricing agent never received the regional cap" is a mechanism. "The cap had no owner in the task and rule-attribution maps" is an actionable root cause.

Grade evidence before assigning causality. Direct evidence comes from the trace, such as the exact payload an agent received. Corroborating evidence comes from constraint results, tool logs, or a repeated pattern across runs. Inferred evidence fills a gap and must be labeled as such. A confident narrative built on an unobserved handoff is still an inference.

Diagnosis also needs a stopping rule. Continue moving upstream while the current explanation depends on another observed decision. Stop when the team reaches a condition it owns and can test directly. Stopping at "the model reasoned badly" is usually premature because model behavior was shaped by the task, information, tools, and controls supplied by the system.

A shared taxonomy makes diagnosis repeatable. Decomposition faults assign the wrong work to a role. Constraint blindness leaves a required rule outside an agent's decision context. Coordination breakdowns lose or distort information between roles. Optimization failures consume unacceptable time, tokens, or calls.

Hallucinated dependencies introduce a resource, prerequisite, or authority that does not exist. Information boundary violations expose data to a role that should not receive it. Temporal misalignment performs valid actions in an invalid order or with stale state. The seven-pattern field guide defines observable signals and prevention controls for each category.

The production taxonomy also supports contract violations as an interface-specific extension. Contract testing deserves separate treatment because it evaluates an explicit producer-consumer boundary. The agent contract testing guide explains that layer.

A causal chain is a directed graph. Each node is an evidenced event or classified fault. Each edge states that one event materially enabled another. Root causes have no incoming causal edge inside the observed run. Symptoms sit downstream.

Chronology alone is not causality. If a retrieval call happened before an invalid approval, it is not automatically the cause. The analyst asks whether the approval would probably have changed if the retrieved evidence had been correct, complete, and available at the decision point.

Start from the violated constraint and walk backward through the decision's inputs. Identify the agent that created each input. Continue until the trace reaches an event that can be changed directly, such as a missing rule, ambiguous task assignment, stale tool response, or unvalidated contract. The causal-chain debugging guide turns this method into a repeatable worksheet.

Build the graph in two passes. The first pass adds only high-confidence edges supported by trace evidence. The second pass adds plausible edges that require replay or human review. Keeping those sets separate prevents a polished explanation from hiding uncertainty.

Use three tests for a proposed edge. Temporal precedence asks whether the cause existed before the effect. Mechanism asks whether there is a credible path by which it changed the downstream decision. Counterfactual dependence asks whether removing or correcting it should change that decision. An edge that fails any test should remain a hypothesis.

Shared causes matter more than long chains. If five downstream decisions all depend on the same incomplete handoff, the graph should converge on that boundary. Treating each visible violation as a separate root creates five tickets and leaves the common control gap untouched.

A counterfactual states the smallest intervention expected to break a causal chain. It names a decision point, a concrete change, and the downstream behavior that should differ. "Improve the prompt" is not testable. "Require the planner to attach the approval threshold to every pricing task" is.

Replay validates the claim. Run the same scenario and initial state with the intervention applied. Keep unrelated variables stable where possible. If the targeted chain disappears and no new critical constraint fails, the proposed cause gains support. If the outcome does not change, the diagnosis was incomplete.

Counterfactuals should become regression cases. This connects RCA to the production-ready agent lifecycle: diagnose, change one control, replay, record the evidence, and add the case to the release suite.

Design replay as a small experiment. Preserve the original scenario, initial state, tool fixtures, and constraint set. Change only the proposed control. When model sampling cannot be fixed, run enough repetitions to compare outcome rates rather than treating one improved run as proof.

Measure more than the targeted symptom. Confirm that the original constraint now passes, the suspected causal edge disappears, and latency, cost, escalation, and unrelated critical constraints do not regress. A change that hides one symptom by creating another is not a validated fix.

Prioritize counterfactuals by causal reach, severity, confidence, and implementation cost. A high-confidence boundary check that prevents four downstream faults should usually precede a broad prompt rewrite with uncertain effects. Record the ranking so the remediation plan remains auditable.

Human review remains part of the method. Reviewers should inspect low-confidence root causes, disputed causal edges, and fixes that change authority or escalation. Automation structures the evidence. It does not remove accountability for high-impact changes.

Store the replay as a remediation record. Include the original run, proposed root cause, changed control, expected effect, observed effect, residual uncertainty, and owner. Future incidents can then reuse evidence instead of restarting the same argument from raw logs.

Over time, track which root causes recur and which counterfactuals hold across scenarios. Repeated success turns a one-run fix into an engineering pattern. Repeated failure is a signal that the taxonomy, instrumentation, or causal model needs revision.