Technical analysis
Cross-Run Pattern Detection: Finding Systemic AI Failures
One failed run may be noise. A carefully grouped set of runs can reveal a structural weakness in decomposition, constraints, handoffs, tools, or the model itself.
Primary keyword: AI failure pattern detection · Truvyx Editorial · 12 min read
Why the fifth similar failure matters
An incident review often focuses on the single run that reached a human. That is necessary, but it can hide the scale of the problem. Suppose a delivery-planning agent misses a deadline once because a supplier feed is unavailable. The failure may be incidental. If five runs across three customers all omit the same delivery constraint after a handoff, the system is telling you something structural.
Cross-run analysis turns isolated traces into evidence about a class of behaviour. The useful unit is not just the prompt or the final answer. It may be the same agent, the same handoff schema, a particular tool, a policy version, a time window, or a combination of conditions. Pattern detection helps the team fix the mechanism instead of patching the most recent example.
Normalize runs before grouping them
Runs are difficult to compare when every system names events differently. Create stable fields for scenario, agent, model, tool, policy version, fault type, severity, first divergent event, and outcome. Keep the original trace available, but extract a compact analysis record that can be grouped without reading every transcript.
Normalization should not erase context. A scheduling failure caused by missing timezone information is different from one caused by a tool timeout, even if both end with ‘booking failed’. Preserve the distinction between observed facts and inferred labels. A reviewer should be able to open the underlying trace and challenge a classification when necessary.
Look for concentrations, not just counts
A count tells you that a failure happened. A concentration tells you where to intervene. Break failures down by agent, scenario family, model version, tool, constraint, and handoff. Then look for intersections. Perhaps constraint violations are rare overall but cluster whenever the planner delegates to a particular specialist. Perhaps latency spikes only when a fallback model is used after a retrieval miss.
Imagine 200 runs with 12 failures. The headline rate is reassuring. A cross-tab reveals that 10 of those failures occur in the 20 cases involving a high-value customer, and all happen after the same context compaction step. The aggregate percentage hid the operational risk; the pattern made it visible.
Separate recurring causes from recurring symptoms
The same visible symptom can have several causes. ‘Wrong answer’ may reflect bad decomposition, stale retrieval, a malformed tool result, a constraint that was never passed downstream, or an evaluator that misunderstood an acceptable alternative. Grouping only by final outcome creates a noisy bucket. Group by the earliest evidence-backed divergence whenever possible.
This is where a failure taxonomy becomes useful. Labels such as coordination breakdown, constraint blindness, hallucinated dependency, and temporal misalignment give teams a shared language, but the label is a starting point rather than a verdict. Link every label to trace evidence and a proposed next experiment: change the contract, add a verifier, alter retrieval, or replay with a controlled tool response.
Close the loop with prevention
A detected pattern should produce an engineering action and a new regression scenario. If five runs show that a handoff drops a customer’s approval status, add a contract check and a scenario that fails when the field is absent. If a model update increases unnecessary tool loops, add a cost or trajectory-length threshold and route the affected task differently.
The most valuable outcome is not a beautiful chart. It is a smaller chance that the same class of failure returns unnoticed. Monitoring can surface the pattern, execution traces can explain it, RCA can test the causal hypothesis, and evaluation can keep the fix in place. That is AI failure pattern detection as a practical reliability discipline.