Designing a Constraint Engine for AI Agents

Every constraint should have an identifier, severity, owner, scope, version, and evaluation method. “Do not disclose a patient identifier” is not enough on its own. The engine also needs to know which fields are identifiers, which agents may receive them, and what evidence proves a disclosure occurred.

Version constraints independently from prompts and models. A policy update should produce a new evaluation baseline, not silently rewrite the meaning of historical runs.

Hard constraints are binary or close to binary: a spend must not exceed a ceiling, a user without approval must not call a destructive tool, and an output must satisfy a schema. Keep these checks deterministic wherever possible.

Soft constraints involve interpretation: whether an explanation is sufficiently clear, whether a plan is proportionate, or whether a response addresses the user’s underlying concern. These can use a calibrated judge, but the result should carry confidence and a review path.

The engine cannot verify what the workflow does not record. Define events for agent turns, handoffs, tool requests, tool results, state changes, approvals, and user-visible outputs. Each event should include a run ID, timestamp, actor, payload reference, and version.

Prefer references to sensitive payloads over copying raw data into the constraint service. The engine can evaluate a field classification or a hash-backed record while the source system retains the protected value.

A practical order is: schema validation, identity and permission checks, temporal assertions, domain rules, semantic judges, then aggregation. Cheap and decisive checks should run early. A failed permission gate should prevent an expensive semantic evaluation from being mistaken for an approval.

Return structured results such as { "constraint": "vendor.approved", "status": "failed", "evidence": ["event_18"] }. Evidence is the product. A green or red badge without the event, field, or comparison that produced it does not help an engineer fix the system.

A constraint engine should distinguish pass, fail, and unknown. Missing telemetry is not proof of compliance. If a tool call has no recorded authorization decision, the engine should return unknown or fail according to the constraint’s severity.

This matters operationally. Teams may accept an unknown for a low-risk recommendation but require a hard failure for a regulated decision. The policy—not the implementation accident—should determine the response.

Store the evaluated expression, the relevant events, the constraint version, and the inputs used for the decision. This makes a result reproducible when a policy changes or a model is upgraded.

Add counterfactual replay: remove the offending tool call, restore the missing approval, or substitute the expected handoff and run the checks again. If one change clears several violations, it is a strong candidate for the root fix.

The first mistake is putting every rule in a prompt. The second is building one giant evaluator that returns a single score. The third is logging the final answer but not the path. The fourth is allowing policy authors to change rules without versioning or review.

A smaller engine with clear contracts is safer: explicit constraints, typed events, deterministic primitives, calibrated judgement, and signed results. Expand its vocabulary as real incidents justify it.

A strong AI constraint engine does not try to make an agent infallible. It makes important rules observable and enforceable. It can tell a developer which event violated which policy, an auditor which version was applied, and an operator whether the system should stop, escalate, or continue.