Interface reliability
Agent Contract Testing: Catching Breaking Changes Before They Reach Production
Every agent handoff contains a promise. The producer promises a shape, meaning, level of completeness, and timing. The consumer builds its next decision on that promise, whether or not anyone documented it.
Agent contract testing makes the promise executable. It applies the system perspective from multi-agent evaluation to each producer-consumer edge.
What inter-agent contract testing is
A contract defines what a consumer requires from a producer. Structural terms include required fields, types, formats, enum values, and nesting. Semantic terms include units, definitions, confidence meaning, source expectations, and freshness.
The consumer owns the minimum it needs. The producer proves that its output satisfies those needs. This consumer-driven model avoids forcing every downstream role to depend on the producer's entire internal representation.
Contracts should be versioned artifacts. A test run records the producer version, consumer version, contract version, observed payload, and each violation. The result can block a release or warn about a compatible extension.
Why breaking changes often go unnoticed
Teams test agents individually. A producer's new output can look better in isolation while becoming incompatible with its consumer. Natural language makes this harder because plausible text can conceal missing structure.
Schema compatibility is necessary but incomplete. Renaming a field causes an obvious break. Reversing a scale, changing currency units, broadening an enum, or replacing verified facts with inferred claims can preserve valid JSON while changing meaning.
Loose parsing delays detection. A consumer may default a missing value, coerce a type, or continue with partial data. The visible problem appears several steps later, so the upstream interface change escapes blame.
The step-level evaluation method preserves the handoff evidence needed to detect the first incompatible transition.
How contract inference works
When no explicit contract exists, infer a draft from successful historical handoffs. Group outputs by producer-consumer edge. Identify stable required fields, observed types, enum sets, numeric ranges, formats, and nested structures.
Inference proposes a contract. It does not establish business truth. Engineers review whether frequent fields are genuinely required, whether rare values are valid, and whether historical runs omitted an obligation that should exist.
Add semantic requirements that structure cannot express. Document units, field definitions, confidence calibration, authoritative sources, maximum age, and how absence differs from zero or false.
Classify compatibility. Removing or changing the type of a required field is breaking. Expanding an allowed range may be degraded if the consumer was calibrated on the old range. Adding an optional field is usually compatible. Use representative fixtures to confirm the classification.
The cost of upstream changes breaking downstream agents
A broken boundary rarely stops at one consumer. It can alter decisions, trigger unnecessary tool calls, corrupt stored state, and create misleading audit evidence. The later the defect is detected, the more downstream artifacts must be unwound.
Run contract tests in CI whenever a producer prompt, model, parser, schema, or tool adapter changes. Test recorded fixtures plus edge values. Run the affected system scenarios after the edge passes, because local compatibility does not prove global behavior.
When a contract violation reaches a run, propagate it into root-cause analysis as the first evidenced boundary defect. The failure-pattern taxonomy then distinguishes the interface break from broader coordination or decomposition problems.
The goal is not to freeze agent outputs. It is to let producers evolve without surprising consumers. Explicit versioning, compatibility rules, and targeted system replay make that evolution safe.