How to Capture and Analyse Multi-Agent Execution Traces

A trace is more than a transcript

When a single chatbot gives an unexpected answer, a transcript may be enough to begin debugging. A multi-agent system is different. The visible conversation is only the surface of the run. Underneath it, a planner created a task, a specialist received a narrowed context, a tool returned data, another agent interpreted that data, and a policy check either allowed or rejected the next action. If those events are flattened into one chat log, the moment the workflow went wrong becomes hard to find.

Think about a travel assistant arranging a complicated itinerary. The planner selects a flight, the policy agent checks the budget, the booking agent calls a supplier, and the notification agent tells the customer what happened. A final itinerary can look reasonable even if the booking agent used an expired fare or the notification agent announced a reservation that was never confirmed. An execution trace preserves the difference between intention, attempted action, confirmed state, and reported outcome.

Choose the events your future self will need

Start with a small event vocabulary rather than logging every internal detail. A useful event normally has a run identifier, timestamp, agent identity, parent event, input reference, output reference, tool or model name, and resulting state. The parent relationship matters because it lets you reconstruct causality: the verification call happened because the planner classified the request as high risk, not merely because it appeared a few lines later.

Do not record secrets simply because tracing feels like observability. Store references to sensitive payloads, redact credentials, and make the retention policy part of the design. For a healthcare scheduling workflow, the trace might retain that a patient record was consulted and that a clinical constraint blocked a slot, while keeping the actual protected fields behind an access-controlled evidence store.

type TraceEvent = {
  runId: string;
  eventId: string;
  parentId?: string;
  agent: string;
  kind: "decision" | "tool" | "handoff" | "check";
  inputRef?: string;
  outputRef?: string;
  state: "attempted" | "confirmed" | "blocked";
};

Make handoffs and state changes explicit

Most multi-agent failures hide in the handoff. Agent A may send a summary to Agent B, but the summary can omit a constraint, change the meaning of a date, or present a recommendation as if it were an approved decision. A trace should therefore record the handoff as its own event, with the schema or contract that governed it and the context that was intentionally withheld.

State changes deserve the same treatment. If an agent updates a ticket, reserves inventory, sends a message, or changes a permission, record the attempted state transition and the external confirmation separately. This distinction gives an investigator a precise answer to a common incident question: did the agent merely propose an action, or did the system actually perform it?

Analyse a trace from the outside in

When a run fails, begin at the observed outcome and walk backward through the trace. Locate the first event that made the final outcome inevitable. In the itinerary example, the customer-facing message is not necessarily the cause; the first decisive event may be a supplier response that was treated as a booking confirmation even though it was only a price quote.

This is where execution tracing connects to root-cause analysis. A good trace lets you compare a failed run with a passing run, identify the earliest divergence, and create a scenario that protects the newly discovered boundary. The trace is not an archive that engineers read only during an incident. It becomes training material for better scenarios, contracts, verifiers, and regression suites.

Turn traces into an engineering habit

Begin by tracing one important workflow end to end. Capture the planner, each specialist, every tool call, each hard check, and the final state. Ask someone who did not build the workflow to explain the run from the trace alone. If they cannot tell what was known, decided, attempted, and confirmed, the trace needs another event or a clearer relationship.

Once the shape is useful, connect it to evaluation. Scenario Studio supplies the situation, the run produces the trace, Verifier Factory checks the observable requirements, and RCA Engine explains failures. That loop is how agent execution tracing becomes part of production quality rather than another dashboard that nobody opens.