CI/CD implementation
How to Implement CI/CD Gates for Multi-Agent Systems (With Python SDK Examples)
A CI gate converts evaluation from a report into a release decision. The pipeline runs the agent system against versioned scenarios, checks every hard constraint, compares quality and cost with a baseline, and returns a nonzero exit code when the release is unsafe.
This guide implements the lifecycle described in production-ready AI agents using the concepts defined in the evaluation glossary.
Why every code change needs automated evaluation
Agent behavior changes when orchestration code changes, but prompts, schemas, model versions, retrieval content, tools, permissions, and policies can have the same effect. Treat all of them as release inputs and record their versions.
Map each change to an impact surface. A parser change affects consumers of its schema. A pricing-policy change affects scenarios with price decisions. A shared model change can require the broad suite. This keeps routine pull requests fast without leaving relevant workflows untested.
Run deterministic checks before model calls. Validate configuration, schemas, contracts, permissions, constraint syntax, and scenario feasibility first. Fail quickly when the release cannot possibly pass.
Use stable fixtures for external tools in pull-request evaluation. Live APIs add network noise, mutable data, and side effects. Run a separate integration stage against controlled test environments when tool compatibility itself is under review.
Repeat stochastic scenarios according to risk. One run may be enough for deterministic formatting. High-impact reasoning should use multiple trials and evaluate distributions, worst cases, and recurring fault categories.
The system-level principles in multi-agent evaluation explain why a final-output assertion alone misses contracts, tool order, information boundaries, and cost.
Integrating Truvyx into GitHub Actions / GitLab CI
Create a scoped API key with run submission permissions and store it in the CI secret manager. Never place it in repository variables, workflow output, fixtures, or logs. Use separate keys for development, staging, and production gates.
Keep scenario IDs and required thresholds in version-controlled gate configuration. The scenario content remains versioned in Truvyx, while the repository records which suite and policy the release expects.
The Python SDK accepts an agent function, fetches the scenario, runs the function, submits output and optional trace evidence, polls for a result, and returns a typed verdict. A minimal blocking script looks like this:
Polling for verdicts and blocking deployment on regressions
Evaluation is asynchronous because agent execution and verification can take longer than an ordinary unit test. Poll with a bounded timeout and a fixed or backoff interval. Treat timeout as an infrastructure outcome, not as a quality pass.
Distinguish terminal states. PASS permits the next stage. FAIL blocks on evaluated behavior. ERROR indicates evaluator or infrastructure failure. CANCELED indicates an explicit stop. Your policy should state whether infrastructure errors block, retry, or require approval.
Evaluate per-constraint results before aggregate scores. A critical permission or regulatory failure blocks even when the overall score is high. Then apply minimum semantic-quality, latency, and cost thresholds.
Compare with a baseline. A candidate can remain above the absolute minimum and still regress materially. Store score deltas by scenario and category, not only a single suite average.
Publish a concise CI summary with run links, changed dimensions, critical violations, and artifact identifiers. Keep raw traces and secrets out of public pull-request comments.
The regression monitoring guide extends this known-change gate to production changes that have no corresponding commit or deployment event.
Real-world example of successful CI integration
Consider a customer operations system with triage, account, policy, and response agents. A pull request changes the response agent's prompt and its handoff schema. The impact map selects contract tests plus refund, cancellation, and escalation scenarios.
The first stage validates schemas and policy constraints without model calls. The second runs recorded handoff fixtures. The third executes the selected scenarios several times with frozen tool responses. The final stage compares correctness, escalation, latency, and cost with the main-branch baseline.
A compatibility warning may permit review when an optional field was added. A missing approval field blocks immediately. A small style decline may warn, while any false-autonomy event blocks. The policy reflects consequence rather than treating all metrics equally.
When the gate fails, the run link shows the violated constraint and trace step. The engineer changes one control, reruns the same fixtures, and preserves both results. The pull request becomes an audit record of the risk, evidence, and remediation.
After merge, deploy to a controlled environment and run live-tool integration scenarios. Promote only after those checks pass. Continue production sampling because real users and silent provider updates create conditions the CI suite cannot observe.
Review gate effectiveness quarterly. Track escaped defects, noisy rules, override frequency, runtime, and cost. Remove redundant cases, strengthen missing boundaries, and keep the fastest useful checks closest to the developer.
Use branch protection to require the evaluation job, but keep the workflow definition under review like production code. Pin action versions, restrict token permissions, and prevent untrusted pull requests from reading evaluation secrets.
Control concurrency and rate limits. Bound parallel model calls, queue broad suites, and cancel superseded runs on the same branch. Faster feedback is useful only when it does not create provider throttling or inconsistent partial results.
Cache only immutable inputs and deterministic results. Never reuse a semantic verdict across prompt, model, rubric, scenario, or source-version changes. Include those identifiers in every cache key.
Treat flaky evaluation as a defect. Measure rerun agreement, isolate infrastructure errors from behavioral variance, and use repeated trials where randomness is expected. Do not normalize repeatedly clicking rerun until a release passes.
Maintain separate policies for pull requests, staging, and production promotion. Pull requests emphasize targeted speed. Staging adds live-tool compatibility and broader suites. Promotion requires the strongest evidence and explicit override controls.
Export machine-readable results as artifacts so later jobs can enforce policy without scraping console text. Preserve human summaries for review, but let structured constraint and score fields drive automation.
Add a scheduled full-suite run even when change-based selection is used on pull requests. Dependency maps can be incomplete, and shared external changes may affect workflows no commit touched.
Define artifact retention by risk and audit need. Keep verdicts, version identifiers, constraint evidence, and approvals long enough to reconstruct a release without retaining unnecessary sensitive payloads.
Test the gate configuration itself with known passing, failing, timeout, and infrastructure-error fixtures. A pipeline that cannot prove it blocks the right states should not be trusted as a release control.
Give developers a local or preview path that uses the same gate configuration as CI. Fast pre-push feedback reduces avoidable remote runs and makes a blocked pipeline easier to reproduce.