Why Multi-Agent Evaluation Is Hard

Multi-agent systems are not just harder to build than single-agent ones. They are harder to evaluate in a fundamentally different way. A single LLM agent either answers correctly or it does not. A multi-agent system can have every individual component pass its own tests and still fail the task, fail its users, and fail its compliance requirements, all at once.

The single-agent benchmark assumption

Standard AI benchmarks, MMLU, HumanEval, HellaSwag, GSM8K, and most datasets on the Open LLM Leaderboard, measure isolated capability. They present a prompt to a model, collect a response, and score it against a reference answer.

That model works when:

• One model is responsible for the entire output
• The input is complete at inference time
• There is no coordination layer between components
• Failure is visible in the final output

Multi-agent systems break all four of these assumptions at once.

In a typical operational AI system, you might have a planning agent that decomposes the task, specialist agents that handle subproblems, a synthesis agent that assembles the final output, and a verification agent that checks the result. Each of these agents can score 95% on isolated capability benchmarks. The composed system might still fail 35% of the time in production, not because any one agent is weak, but because agents fail to coordinate.

What actually goes wrong

There are several distinct failure modes in multi-agent systems that single-agent evaluation will never surface.

Constraint blindness

Each agent may operate within its own instructions while violating a system-level constraint that no single agent was explicitly told to enforce. A financial analysis agent might correctly compute a risk score. A reporting agent might correctly format a document. Neither agent is wrong in isolation, but together they produce a report that surfaces a regulatory figure the organisation is not permitted to disclose to a particular counterparty.

Constraint blindness is not a reasoning failure. It is an architecture failure. The constraint was never expressed at the boundary where it needed to be enforced.

Coordination breakdown

Multi-agent systems pass data, context, and decisions across agent boundaries. When those handoffs are poorly specified, the receiving agent gets less context than it needs, or inconsistent context from multiple sources. Both produce failures that look like reasoning failures when the root cause is architectural.

Information boundary violations

In regulated domains, agents are often required not to access or surface certain fields, regardless of how those fields would affect output quality. A model that performs better with access to protected data will, over time, find ways to extract or infer that data if the boundary is enforced only at the prompt level and not verified at evaluation time.

Redundant calls and optimization failure

Agent systems frequently develop loops: a verification agent calls a reasoning agent, which triggers the verification agent again. Each individual call is correct. The loop is not. A system that reaches the right answer via 40 API calls when 4 would suffice is failing an operational requirement, even if the benchmark gives it full marks.

Attribution failure

When a decision is wrong in a multi-agent system, which agent made it? If that question cannot be answered after the run, you cannot fix the problem. You can only re-run and hope for a different outcome. Attribution failure is why many teams end up in expensive manual review loops instead of directed engineering work.

Contract violations

When Agent A's output is the input for Agent B, there is an implicit contract about schema, format, and semantics. When that contract breaks, Agent B fails in ways that look like reasoning errors but are actually interface errors. You do not fix an interface problem by improving a reasoning model.

Why emergent failures are so hard to catch

The interaction effects between agents are non-linear. A change that makes Agent A 5% more accurate can make the overall system 20% less reliable if it shifts the distribution of Agent A's outputs in a way that Agent B was not designed to handle. Isolated benchmarking will attribute the improvement to Agent A and never surface the system-level regression.

Non-determinism compounds at every handoff. LLMs produce different outputs for the same input across runs. In a single-agent setting you can run a benchmark multiple times and report variance. In a multi-agent setting, the outputs of earlier agents become the inputs for later agents. Variance compounds at each step. A system that passes evaluation on Monday can fail on Friday, not because anything was deployed, but because a provider silently updated a model behind the same API endpoint.

State accumulation is a third factor. Many multi-agent systems maintain context across turns in a session. Failures in turn 3 may only become visible in turn 7, and the causal trace connecting them is buried in a 200-message context window that no one is going to read manually.

What evaluation needs to look like

Multi-agent evaluation requires a different conceptual model than single-agent benchmarking. Four requirements follow directly from the failure modes above.

Scenarios, not prompts.

A scenario describes what the multi-agent system is supposed to accomplish, in enough detail that constraints can be checked and outcomes can be verified without a reference answer. Scenarios define agent roles, input parameters, and the specific conditions under which the run counts as a pass or a failure. The scenario is the unit of evaluation, not the individual prompt.

Constraint-level granularity.

Aggregate scores hide failure modes. A system that scores 0.88 overall might be failing 100% of the time on one constraint category and compensating with strong performance on others. Evaluation must report per-constraint pass/fail. A score of 0.88 on a compliance-critical constraint is a failed system, regardless of the aggregate.

Observability into the coordination layer.

You need the full trace: which agent produced what output, in what order, with what inputs. Without this, root-cause analysis is guesswork. The trace is the evidence. Evaluation infrastructure that does not capture the trace is discarding the most valuable diagnostic data it has.

Constraint verification as first-class evaluation.

Constraints should be expressed in a structure formal enough that a constraint engine can evaluate them automatically against the agent output, without requiring human judgment for every run. This is what makes evaluation repeatable across model updates and deployable in CI pipelines.

Knowing something broke is not enough

Even with proper scenario-level evaluation and per-constraint reporting, a pass/fail result only tells you that something went wrong. It does not tell you where, why, or what to change. That requires a separate capability: root-cause analysis.

The next post in this series covers how RCA works for multi-agent systems, including the six-stage pipeline from trace processing to counterfactual generation that turns evaluation results into scoped engineering tasks.