The Definitive Glossary of Multi-Agent Evaluation Terms (2026 Edition)

Multi-agent evaluation needs precise language because vague language produces vague controls. Teams cannot fix constraint blindness, contract drift, or attribution failure if those terms are not defined consistently across engineering, product, security, and compliance.

Use this glossary as the companion reference to What is Multi-Agent Evaluation? and as a shared vocabulary for evaluation design reviews.

Alphabetical glossary of core terms

These terms map to real engineering objects. A scenario should exist in a test suite. A trace should exist as a stored artifact. A constraint should be machine-checkable where possible. A verifier should produce evidence, not just a confidence score.

Fault type definitions

Fault types turn failures into diagnosable categories. The Truvyx RCA taxonomy is explained in more depth in Root-Cause Analysis for Agents, but the definitions below are the working vocabulary every evaluation suite should use.

Fault labels should not be treated as blame labels. A coordination breakdown does not mean one agent is bad. It usually means the boundary between agents is underspecified. A constraint blindness fault often points to missing architecture, not a weak prompt.

For worked examples of these categories across decomposition, coordination, information access, and timing, use the seven multi-agent failure patterns. A shared taxonomy makes incident counts comparable across teams and across evaluation runs.

Constraint and reliability terms

Hard constraint.

A rule that must not be violated. Examples include budget ceilings, prohibited fields, approval requirements, role restrictions, and temporal ordering rules.

Soft constraint.

A preference or quality target. Examples include concise output, helpful tone, response style, or evidence richness. Soft constraints can be scored, but they usually do not block execution by themselves.

Provable feasibility.

Evidence that a scenario can satisfy its declared constraints before or during execution. This matters because a system can fail not due to model weakness, but because the task was impossible under its own rules.

Constraint engine.

A system that encodes and evaluates rules against agent outputs, traces, or scenario specifications. The Truvyx docs explain how scenarios, constraints, and submitted agent outputs fit together in the product.

Evaluation run.

One execution of a scenario against an agent system output, including scores, violations, trace evidence, and status.

Baseline.

A known reference state used to detect regressions. Baselines can be established from a previous model version, a production run set, or a curated scenario suite.

These concepts work as a system. The four pillars of agentic reliability connect scenario coverage and hard constraints to causal traces and regression monitoring.

Lifecycle and CI/CD terms

The production-ready agent guide places these terms in release order, from pre-deployment scenarios and quality gates to monitored production runs.

Evaluation gate.

A CI or release-control step that blocks deployment when a scenario suite falls below defined thresholds.

Regression monitoring.

Continuous or scheduled evaluation designed to catch silent degradation after model, prompt, data, or tool changes.

Counterfactual.

A minimal intervention that would have prevented a failure. A good counterfactual names the agent, boundary, or rule that should change, plus the downstream faults that change would prevent.

Maturity model.

A staged framework for assessing how reliable an organization's agent evaluation practice is. The multi-agent evaluation maturity model moves from manual prompt testing to automated constraint-proven CI/CD.

The practical test for this vocabulary is simple. If a team can point to the artifact behind each word, it has a foundation for evaluation. If the words exist only in meetings, the system is still being judged by impressions.