Designing a Constraint Engine for AI Agents

A constraint engine sits beside an agent workflow. It does not need to make the agent intelligent. Its job is narrower and more valuable: decide whether the workflow stayed within a set of declared rules.

For example, an operations agent may be allowed to restart a service only when three conditions are true: an incident owner approved the action, a recent backup exists, and the service is not under a change freeze. The agent can propose the restart. The constraint engine checks the evidence.

This separation is important. The component that wants to act should not be the only component that decides whether the action was allowed.

Avoid storing rules as loose sentences in different prompts and code files. Give each constraint an ID, a name, a description, a severity, an owner, a scope, a version, and an evaluation method.

A constraint object might say that an agent may call a refund tool only when the order is eligible and the amount is below its approval limit. The object should also explain what evidence proves compliance: an eligibility record, the caller’s role, and the requested amount.

Version constraints independently from models and prompts. When a policy changes, create a new version so old evaluation runs still mean what they meant when they were executed.

Natural language is excellent for helping people author and understand policy. It is not always precise enough to enforce policy by itself.

A useful design keeps two layers. The human-readable layer explains the rule: “Only an approved manager may authorise a refund above $500.” The executable layer defines the fields, comparison, role lookup, and evidence required to test it.

You can use an LLM to help translate a draft policy into a candidate constraint, but a person should review the result before it becomes a hard gate. The translation is assistance, not authority.

A constraint engine can only check what the workflow records. Define a small event vocabulary for agent turns, handoffs, tool requests, tool results, state changes, approvals, and user-visible outputs.

Each event should include a run ID, timestamp, actor, event type, payload reference, and relevant version. For sensitive payloads, prefer references, classifications, or redacted copies over duplicating the raw data.

For example, a tool-call event might include the tool name, the caller’s role, the target resource, the fields sent, and the approval ID. That is enough to evaluate many access constraints without copying a customer record into every log.

A practical evaluation order is simple checks first, expensive interpretation later. Start with schema validation. Then check identity and permissions, event order, domain rules, and finally semantic quality.

A schema check can confirm that an approval ID is present. A permission check can verify that the approver has the right role. A temporal check can confirm that approval happened before the write. An LLM judge might assess whether the explanation clearly describes the risk.

Do not use an LLM judge for arithmetic, permissions, or exact event ordering when ordinary code can answer the question. Deterministic checks are faster, easier to reproduce, and easier to audit.

A red badge tells a developer that something went wrong. Evidence tells them what to fix.

Return the constraint ID, status, severity, evaluated version, relevant event IDs, and a short explanation. A result might say that the approval was missing, point to the tool-call event, and identify the agent that attempted the action.

A structured result can look like this:

{ "constraint": "refund.approval", "status": "failed", "evidence": ["event_18"], "severity": "high" }

The evidence also helps auditors and operators. They can see which policy version ran and whether the result was based on a complete trace.

One of the most dangerous design choices is to treat missing evidence as a pass. If the workflow did not record an authorization decision, the engine cannot honestly say authorization was valid.

Use at least three outcomes: pass, fail, and unknown. Then let policy determine what unknown means. A low-risk recommendation may continue with a warning. A regulated payment may require a stop and human review.

This distinction prevents observability gaps from becoming accidental approvals.

Some constraints belong before an action, some after it, and some in both places. A destructive tool should be checked before execution. A data-leak rule should inspect the outgoing payload before transmission. A result-quality rule may run after the workflow completes.

For high-severity actions, use defence in depth. The agent can be instructed not to act, the tool gateway can check permissions, and the evaluator can verify what happened. No single prompt or service should be the only line of defence.

Store the evaluated expression, constraint version, relevant inputs, and selected events. This lets you reproduce a result when the model, policy, or data changes.

Then add counterfactual replay. Remove the invalid tool call, restore the missing approval, or substitute the correct handoff and run the checks again. If one change clears several failures, it is a strong candidate for the root fix.

Replay also prevents teams from “fixing” a problem by changing five things at once and losing track of what actually helped.

The first mistake is putting every rule in a prompt. Prompts guide behaviour; they do not prove behaviour.

The second is building one giant evaluator that returns one score. A single score hides which boundary failed and makes release decisions difficult.

The third is recording only the final answer. Without events, the engine cannot distinguish a safe result from a lucky result.

The fourth is allowing policy changes without review or versioning. If a rule can change silently, historical comparisons become unreliable.

Start with ten important constraints rather than a language for every possible policy. Choose rules that are high-impact and observable: permission checks, budget ceilings, required approvals, prohibited fields, valid output schemas, and tool order.

Give each one a deterministic evaluator, a clear owner, and a test scenario that should pass and one that should fail. Add semantic judging only where the rule genuinely requires interpretation.

A good AI constraint engine does not try to make an agent perfect. It makes important boundaries visible, testable, and difficult to bypass.