Evaluation methods
Step-Level Evaluation: Why Final Output Checks Are Not Enough
A correct final answer is not proof of a reliable agent system. The system may have used prohibited data, skipped an approval, called the wrong tool, or reached the result through a loop that will not survive the next input.
Step-level evaluation scores the decisions and transitions inside a trajectory. It complements the system view described in the multi-agent evaluation guide.
What step-level evaluation is
A step is an observable transition: an agent decision, delegation, tool call, handoff, approval, or state mutation. The evaluator records the input state, action, output state, applicable constraints, and evidence used.
Each step receives only the checks relevant to it. A retrieval step can be judged on source authority and freshness. A calculation step can be recomputed. A handoff can be checked against a contract. An approval can be checked against role and threshold rules.
This creates a layered verdict. The final outcome may pass while one process constraint fails. Another run may miss the preferred outcome while every decision remains safe and explainable. Those cases require different product responses.
The problem of lucky path outputs
A lucky path reaches the expected answer through reasoning or actions that are not dependable. The output hides the defect because the particular input did not expose its consequence.
Consider an agent that calculates a delivery date with the wrong timezone but happens to receive an order placed at midday. The date is correct. The same logic fails near midnight. A final-output test passes the first case. A step check detects the timezone assumption.
Lucky paths also occur when an agent guesses a missing value correctly, retrieves an unofficial source that agrees with the official one, or bypasses approval for an amount that happens to remain below the limit. The process is wrong even when the answer is right.
The causal method in tracing agentic workflows uses these step records to distinguish evidence from coincidence.
How step-level evaluation catches silent failures
Attach invariants to transitions. A role may not read restricted fields. A pricing action must cite the current rate table. A tool call must follow approval. A downstream agent must receive the producer's confidence and source version.
Evaluate both positive and negative obligations. Positive checks confirm required actions happened. Negative checks confirm prohibited actions did not happen. Absence matters: a trace with no recorded approval cannot be treated as approved.
Compare the trace with the scenario's allowed graph. Unexpected edges reveal undeclared delegation or tool use. Missing edges reveal skipped responsibilities. Timing checks expose stale inputs and reordered actions.
Implementation approaches and best practices
Define a stable event schema before adding evaluators. Include run ID, step ID, parent step, agent role, timestamps, input references, output references, tool metadata, state version, and applicable constraint IDs.
Use deterministic checks first. Validate schemas, arithmetic, permissions, required fields, ordering, and known identifiers in code. Use model judgment for semantic questions such as whether a summary preserved a qualification. Route low-confidence or high-impact cases to people.
Keep verdicts attached to evidence. A failed step should identify the rule, observed value, expected condition, and trace location. This makes the result useful to agent RCA instead of creating another opaque score.
Control cost with selective evaluation. Run cheap checks on every step. Run expensive semantic judges only on relevant step types or sampled production traces. Preserve enough raw evidence to replay important cases after evaluators change.