Root-Cause Analysis for Agents

A pass/fail result tells you a system is broken. It does not tell you where, why, or what to change.

In single-service software, root-cause analysis is hard but tractable: you have a stack trace, a log line, a failing test. In multi-agent AI systems, failure is distributed across agent boundaries, compounded by non-determinism, and often visible only several steps downstream from where it originated. The trace exists, but interpreting it requires more than scanning logs.

This post describes how root-cause analysis works for multi-agent systems: what data it requires, how faults are classified, how causal chains are reconstructed, and why counterfactuals produce better engineering tasks than generic recommendations.

Why RCA for agents is structurally different

In traditional software, root-cause analysis starts from an exception or a failing assertion. The failure point is usually the root cause, or close to it.

In multi-agent systems, the point of failure and the root cause are almost never the same thing. Three structural properties make this so.

First, a correctly behaving agent can be the root cause of a downstream failure. An agent that executes its instructions correctly can produce an output that violates a system-level constraint the agent was never told about. The fault registers downstream. The cause is upstream, and it is an architecture decision, not a model error.

Second, faults interact. A coordination failure and a constraint blindness fault can coexist in the same run and reinforce each other. Attributing the final outcome to either one in isolation produces an incomplete diagnosis. You need a representation of the causal relationships between faults, not just a list of them.

Third, the evidence is distributed. An agent output log, a constraint evaluation result, an API call trace, and timing data all contain pieces of the picture. No single artifact tells the full story. RCA requires a pipeline that aggregates and interprets all of these together.

The fault taxonomy

Before you can trace a cause, you need a vocabulary for what went wrong. Multi-agent systems produce a consistent set of fault types across domains and architectures.

These types are not mutually exclusive. A single run can contain multiple fault types, and the causal relationship between them is part of what RCA reconstructs.

The six-stage RCA pipeline

Root-cause analysis runs as a sequential pipeline over the artifacts produced by an evaluation run. Each stage takes the output of the previous one and produces a more structured and actionable representation.

Stage 1: Trace processing

The pipeline starts by decomposing the raw agent output into individual agent decisions with associated metadata: which agent produced the decision, what inputs it received, what outputs it produced, and when. This stage normalises heterogeneous trace formats from different agent frameworks into a common structure. Without normalisation, every subsequent stage would need to understand every possible trace format, which is not maintainable.

The output of trace processing is a sequence of typed agent events with timestamps and dependency relationships. This is the foundation everything else is built on.

Stage 2: Fault classification

Each candidate fault event in the trace is classified into the fault taxonomy using a multi-vote ensemble. The same fault is presented to the classification model three times at a low temperature. The majority label across the three calls becomes the classification. The agreement rate across the three calls becomes the confidence score.

Why an ensemble? Classification tasks benefit from independent draws. A single call that misclassifies a fault registers as 100% confident in the wrong label. The ensemble surfaces disagreement explicitly: a fault that gets two votes for constraint blindness and one for coordination breakdown is genuinely ambiguous, and that ambiguity is useful information. Faults with agreement below 67% (two of three calls agreeing) are flagged for human review rather than being silently assigned a label.

The output of fault classification is a list of fault events, each with a type, a confidence score, and the specific agent decision or boundary crossing that triggered it.

Stage 3: Causal chain reconstruction

The classified faults are assembled into a directed causal graph. Edges in the graph represent causal relationships: fault A caused fault B means that fault B would not have occurred if fault A had not occurred. Root causes are nodes with no incoming edges.

This distinction matters operationally. In a run where a single upstream constraint blindness fault propagates into four downstream coordination failures, the flat fault list shows five faults. The causal graph shows one root cause and four downstream effects. Fixing the root cause eliminates all five. Addressing the four downstream effects independently treats the symptoms and leaves the cause in place.

Causal chain reconstruction is the stage that separates root-cause analysis from fault listing.

Stage 4: Counterfactual generation

For each root cause node in the causal graph, the pipeline generates a counterfactual: the minimal change to agent behaviour or system configuration that would have prevented that fault and its downstream effects.

The word minimal is doing significant work here. A counterfactual that says "redesign your planning agent" is a recommendation, not a counterfactual. A counterfactual that says "adding output schema validation at the boundary between the planning agent and the execution agent would have caught the malformed field that caused the downstream failures" is a scoped task with a verifiable completion criterion.

Counterfactuals are constrained by the pipeline to be single-step interventions. If the causal chain has multiple independent root causes, each gets its own counterfactual. The engineering team can prioritise by expected impact, not by which recommendation sounds most compelling.

Stage 5: Cross-run pattern detection

Individual runs produce individual diagnostics. Patterns across runs produce systemic diagnostics.

A fault type that appears in more than 30% of runs over a 30-day window is surfaced as a systemic weakness. The operational significance of this threshold is that a 30% recurrence rate means the fault is structural, not incidental. Treating a structural fault as a per-run incident is the wrong remediation strategy: you will fix individual instances indefinitely rather than addressing the underlying cause.

Cross-run pattern detection also separates faults that correlate with specific model versions, specific scenario types, or specific time periods from faults that appear across all conditions. This information is necessary for determining whether a fault is caused by a model capability gap (fix: use a different model) or an architecture decision (fix: change the coordination layer).

Stage 6: Summary and remediation plan

The pipeline synthesises findings into an audience-tuned output. Technical teams receive the fault graph, confidence scores, and counterfactuals in structured form. Compliance teams receive a narrative that maps findings to regulatory frameworks. The remediation plan is prioritised by expected impact: highest-confidence root-cause faults that produce the most downstream effects come first.

For regulatory contexts, the summary can be generated as a formal examiner narrative, covering what was tested, what was found, what the remediation plan is, and the evidence supporting each claim. This is a different artifact from the technical output, and it serves a different audience.

What makes a good counterfactual

The difference between a useful counterfactual and a generic recommendation is specificity. Both point at a problem. Only one tells you what to build.

Three properties define a useful counterfactual for multi-agent systems.

It names the specific agent and specific decision point where the intervention would occur. "Improve your prompts" is not a counterfactual. "Agent B's input validation, applied before processing the output from Agent A, would have rejected the malformed field at the boundary rather than propagating it" is a counterfactual.

It specifies the causal path it interrupts. A good counterfactual says which downstream faults the intervention prevents. This is what lets you prioritise: an intervention that prevents one fault is less valuable than one that prevents the same fault plus four others that trace to the same root cause.

It is verifiable. After the intervention is applied, you can re-run the evaluation and confirm that the targeted faults no longer appear. If the counterfactual is too vague to verify, it is a recommendation, not a counterfactual.

Fault patterns that recur across systems

Certain fault patterns appear across multi-agent systems often enough to be treated as predictable properties of the architecture rather than individual bugs.

Verification loops.

An agent that is not confident in its output re-queries an upstream agent, which triggers the first agent again. The loop terminates when a time limit is hit, not when the problem is resolved. The final output is the result of a truncation, not a resolution. Evaluation that scores only the final output will give this system full marks on accuracy while missing the operational failure entirely.

Context accumulation failures.

As sessions grow longer, earlier context becomes progressively less influential relative to recent context. A constraint established in turn 2 of a session can be effectively forgotten by turn 15 as newer context fills the window. Standard evaluation scenarios are typically short enough to avoid this. Production sessions are not.

Silent schema drift.

Agent A is updated and the output schema changes. Agent B, which depends on that output, has not been updated. The mismatch produces no error because the new schema is still valid JSON. The semantic meaning of a field has changed, and Agent B interprets it incorrectly. This fault is invisible to output-level evaluation and requires interface contract testing to surface before it reaches production.

Cascade constraint violations.

A single constraint violation early in a run propagates forward through the agent graph. Each subsequent agent makes locally reasonable decisions given its inputs. The final output violates multiple constraints. A flat violation list treats this as five separate problems. A causal graph shows it as one root cause with four downstream effects.

The data requirements

RCA is only as good as the data available to it. There are four data requirements for the pipeline to produce reliable results.

Agent decisions must be captured individually, not just aggregated into a final output. This means instrumentation at the agent level, not just at the system output level.

Constraint evaluation must be per-constraint, not just an aggregate score. The fault classification and causal chain reconstruction stages need to know which specific constraint was violated, not that the overall score was 0.72.

API call traces, including timing, are required for redundant call detection and loop detection. Without timestamps, optimization failures are invisible.

Enough historical runs must be available for cross-run pattern detection to be statistically meaningful. The 30% recurrence threshold is not useful with two runs. It requires enough volume that a 30% rate is distinguishable from noise.

Most teams discover that their existing logging infrastructure captures some but not all of these. The missing pieces are typically per-agent decision capture and per-constraint violation records, which require intentional instrumentation at the evaluation design stage rather than retrofitting after the fact.

What RCA produces

A well-executed root-cause analysis run for a multi-agent system produces three things.

A ranked list of root causes, each with a confidence score derived from the ensemble agreement rate and a count of downstream faults it caused. This tells the engineering team where to direct effort first.

A counterfactual for each root cause, specifying the minimal intervention and the downstream faults it would prevent. This turns the ranked list into a prioritised task backlog.

A cross-run pattern summary, separating one-time faults from recurring structural weaknesses. This tells the team which items are immediate fixes and which require architectural decisions.

Together, these three outputs replace the loop of "run evaluation, see something failed, guess at the cause, make a change, re-run" with a structured engineering process that produces measurable progress per sprint.