Introduction
Truvyx User Guide
Truvyx is an evaluation platform for multi-agent AI systems. You submit your agent's outputs against structured test scenarios, and Truvyx tells you whether it satisfied its constraints, why it failed, and how it compares to other systems over time.
Start with Quick start if you're new, or jump straight to any module in the sidebar.
Key concepts
Scenario — a structured test case: task description and a set of constraints the agent must satisfy.
Scenario suite — a versioned collection of scenarios executed and scored as one release gate.
Run — one execution of your agent against a scenario.
Constraint — a rule (HARD = must pass; SOFT = scored) the agent output must satisfy.
Violation — a constraint the agent broke, with a severity level.
More concepts
RCA report — Root Cause Analysis. After a failing run, classifies which category of fault caused it.
Verifier — a generated script that encodes constraint logic for reproducible evaluation.
HITL — Human-in-the-Loop. The agent escalates uncertain decisions to a human reviewer.
Registry — a community catalogue of benchmark results across agent systems.
Getting started
Quick start — first evaluation in five steps
1
Create a scenario
Go to Scenario Studio (/scenarios). Click New scenario, choose a domain, fill in the task description. Save it.
2
Add constraints
Open the Constraint Engine (/constraint-engine). Add at least one HARD constraint — the rule that must pass — and any SOFT constraints for scoring.
3
Submit a run
From /runs, click Submit evaluation. Choose your scenario, paste your agent's output JSON, name the agent system, and submit. The verifier runs automatically.
4
Review results
Click the run to open its detail view. The Results tab shows constraint scores and violations. The Audit Trail tab shows the verifier signature and attribution map.
5
Set up monitoring (optional)
Go to /monitoring and create a config watching your scenario suite. It will alert you via Slack, email, or PagerDuty if future runs regress.
🔑
Connecting your system programmatically? Generate an API key under Settings → API Keys and use the SDK to submit runs from CI. See
API & SDK.
🧩
Working from Claude Code, Cursor, or another coding agent? Skip the API entirely and add the Truvyx MCP server instead. See
MCP server.
Architecture
How all the modules connect
Every evaluation flows through the same backbone. You build scenarios and constraints first, submit runs against them, then use the analysis tools to understand and improve results.
Scenario Studio
→
Constraint Engine
→
Verifier Factory
Runs ↓
RCA Engine
←
Fault Events
→
HITL Analytics
Cost Benchmarks
→
Registry / Leaderboard
→
Monitoring
Slack alerts
Email alerts
PagerDuty
PDF exports
SDK / API
The core loop: Author a scenario → attach constraints → submit a run → review violations → trace root causes in RCA Engine → escalate decisions via HITL → benchmark cost efficiency → publish to Registry → have Monitoring watch for regressions automatically.
Module 01
Scenario Studio
The starting point for all evaluations. A scenario describes what task the agent should perform, in what domain, at what difficulty, and with what expected output shape. Scenarios are reusable — run many agent systems through the same scenario to compare them.
1
Browse existing scenarios
The main list shows all scenarios for your organisation. Filter by domain or difficulty. Click any row to open it.
2
Create a new scenario
Click New scenario. Choose between From Template (pick a pre-built domain template) or From Natural Language. In the Natural Language tab, use the guided intake — answer what your product does, what decisions it makes, and what could go wrong — and Truvyx assembles the problem statement for you.
3
Open a scenario's detail page
Click a scenario to reach /scenarios/[id]. Here you see run history, constraints, verifier status, and can trigger decomposition or contract inference.
4
Decompose the scenario
Click Decompose to open the visual task-decomposition canvas at /scenarios/[id]/decomposition. This breaks the scenario into sub-agent steps.
feeds intoConstraint EngineVerifier FactoryDecomposition DesignerRegistryMonitoring
💡
New to prompt writing? Use the guided intake on the Natural Language tab. Answer four plain-language questions about your product and Truvyx assembles a well-formed problem statement automatically — no prompt-writing experience needed.
Evaluation collections
Scenario Suites
A scenario suite is a named collection of scenarios evaluated together. The suite has a stable identity and immutable versions. Each version pins the scenario revision, verifier revision, order, and critical flag, so an earlier suite run keeps the exact membership it used.
1
Create the suite
Use the REST API or TypeScript/Python SDK to name the suite, add scenario IDs, and mark any release-blocking member as critical.
2
Run your product
The SDK loads the pinned suite version and invokes your supplied agent function once for each scenario, with at most the configured number running concurrently.
3
Submit atomically
The server validates exact membership, reserves quota for the whole suite, and creates the suite run and every child evaluation in one transaction. An Idempotency-Key makes deployment retries safe.
4
Read the aggregate
Poll the suite-run endpoint for the mean score, pass or failure state, critical failures, and links to every child run. Execution errors are reported as ERROR or PARTIAL instead of being counted as passes.
5
Publish after changes
Existing versions keep evaluating their stored snapshots after a source scenario or verifier changes. Publish a new suite version when you want to adopt those changes.
Credentials: the same API key used for individual evaluations supports suites. Use scenarios:read to read, scenarios:write to create or version, runs:write to execute, and runs:read to poll. Product setup issues a one-time connection challenge; presenting it from the connected environment with runs:write records verification evidence. Reconnect and revoke change connection state without changing the key's suite permissions.
↪
Request errors. Invalid JSON returns 400. Valid JSON with invalid fields, duplicates, or missing member outputs returns 422. Idempotency conflicts and incomplete stored snapshots return 409.
Pagination: collection endpoints accept positive page and pageSize values, cap each page at 100 records, and return total-page plus previous/next-page metadata. SDK collection helpers continue until the server reports no next page.
Module 02
Constraint Engine
Constraints are the rules that define what “correct” means for a scenario. HARD constraints must pass (the run fails if violated). SOFT constraints contribute to a score.
1
Add a constraint
Click Add constraint. Choose the scenario, Name, Type (HARD/SOFT), Category, and a natural language description — the plain-English rule the LLM verifier evaluates against.
2
Use a template
Switch to the Templates tab to browse pre-built constraints grouped by category. Click a template to load it, then customise the natural language.
3
Preview CDL
The CDL Preview tab shows the machine-readable Constraint Definition Language representation — useful for debugging or exporting.
4
Restrict sensitive fields
Under Restricted Fields, add output fields that should be Redacted, Blocked, or Audited. Enforced across all runs for your organisation.
| Category | What it covers |
|---|
| Temporal | Deadlines, sequencing, time windows |
| Resource | Token budgets, memory, compute limits |
| Dependency | Agent A must complete before agent B |
| Geographic | Data locality, regional restrictions |
| Regulatory | GDPR, HIPAA, financial compliance rules |
| Safety | Content filters, harm avoidance, access limits |
| Data Governance | Lineage, retention, classification |
| Fairness | Bias, demographic parity |
| Optimization | Cost efficiency, redundancy |
receives fromScenario Studiofeeds intoVerifier FactoryRCA EngineHITL Analytics
Module 03
Verifier Factory
The evaluation engine. Submit a run and the platform checks each constraint and records violations. Compliance mode creates an unsigned audit record; the SDK can sign it locally and the platform marks it verified only after RS256 verification. The run detail page is the central view most teams use day-to-day.
1
Submit a run
Click Submit evaluation. Select a scenario, enter your agent system name, paste or upload the agent output as JSON. The verifier runs and you are taken to the run detail page.
2
Read the Results tab
Shows overall pass/fail status, a score breakdown by constraint, and a list of violations with severity (CRITICAL → HIGH → MEDIUM → LOW). Each violation shows the constraint broken and the agent output that broke it.
3
Inspect the Audit Trail tab
Shows the cryptographic verifier signature, submitter, environment metadata, and a rule-attribution map. Click Generate Examiner Narrative to produce a regulator-ready PDF report.
4
Trigger RCA on failures
If the run has HARD constraint violations, a Run RCA button appears. Clicking it opens the RCA Engine with pre-loaded fault events for this run.
5
Export
Download results as JSON, or the Examiner Narrative as PDF/DOCX. Choose the audience type (Central Bank, Healthcare Auditor, Tax Authority, Tribunal, or General Regulator) — the language and regulatory references adapt automatically.
receives fromScenario StudioConstraint Enginefeeds intoRCA EngineHITL AnalyticsCost BenchmarksRegistryMonitoring
Module 04
Decomposition Designer
A visual canvas for mapping task splits across specialised sub-agents — which handles which step, what data flows between them, and where human handoffs happen.
1
Open the canvas
Reach it via /decomposition-designer for a blank canvas, or from a scenario detail page via Decompose which pre-loads the scenario context.
2
Add agents and connect them
Drag agents onto the canvas, name them, assign roles. Connect agents with edges representing data flow or control flow.
3
Mark escalation points
Where an agent should escalate to a human, mark the edge as a HITL handoff. This feeds the HITL Analytics module with baseline expectations.
4
Infer contracts
Once saved, navigate to the scenario's contracts view and click Infer contracts to automatically derive the output schema each producer agent hands to its consumer. These become inter-agent contract tests.
receives fromScenario Studiofeeds intoHITL AnalyticsVerifier Factory
Module 05
Registry
A shared catalogue of benchmark results. When you publish a run, it enters the Registry. The Leaderboard ranks agent systems by performance within a domain. Use it to compare against others, find existing scenarios, or contribute results to the community.
1
Browse entries
Filter by domain, difficulty, or agent system name. Click any entry for details. Each published entry has a permanent URL at /registry/[slug] you can share externally.
2
View the Leaderboard
Navigate to /registry/leaderboard. Sort by overall score, HARD constraint pass rate, cost-per-correct-decision, or efficiency percentile. Filter by scenario or domain for a like-for-like comparison.
3
Contribute a result
Go to /registry/contribute. Select a run and confirm publication. Published runs include agent system name, scenario, scores, violation counts, and verifier checksum — no raw agent output is shared.
receives fromVerifier FactoryCost Benchmarksfeeds intoMonitoring (baselines)
Module 06
RCA Engine
When runs fail, Root Cause Analysis classifies why. The engine collects fault events, groups them by fault type, identifies systemic weaknesses across multiple runs, and surfaces recurring patterns.
| Fault type | What it means |
|---|
| Decomposition Fault | The task split between agents was incorrect or incomplete |
| Constraint Blindness | The agent acted without checking a constraint it should have |
| Coordination Breakdown | Agents disagreed or handed off data in an incompatible format |
| Optimization Failure | The agent chose a locally optimal path that violates a global constraint |
| Hallucinated Dependency | The agent assumed a dependency that does not exist |
| Information Boundary Violation | Agent accessed or leaked data outside its authorised scope |
| Temporal Misalignment | Actions taken out of sequence or outside time windows |
| Unknown | Pattern not yet classified — review manually |
1
View cross-run patterns
The RCA Engine home shows fault-type frequency across all your runs. High-frequency fault types indicate a systemic issue with your agent architecture.
2
Open an RCA report from a run
From any failing run's detail page, click Run RCA. This creates an RCA report at /runs/[id]/rca showing fault events, evidence, and root-cause flags.
receives fromVerifier FactoryHITL Analyticsfeeds intoMonitoring
Module 07
HITL Analytics
Tracks escalations — when an agent passes a decision to a human reviewer. Helps you tune escalation thresholds to reduce unnecessary interruptions while keeping humans in the loop where they genuinely add value.
1
View the escalation dashboard
Shows total escalations over time, escalation type breakdown (FALSE_AUTONOMY, UNCERTAINTY, POLICY_CONFLICT, RISK_AVERSION, COORDINATION_BREAKDOWN), approval vs rejection rate, average response time, and semantic drift score.
2
Inspect individual escalations
Each escalation has a decision record: which agent escalated, what action it was considering, who reviewed it, what the human decided (approve / reject / modify), and any corrections.
3
Generate an escalation policy
From a scenario's detail page, click Generate escalation policy. The platform drafts a policy from your constraint definitions and historical escalation patterns. Review and save it.
4
Record an outcome via API
POST to /api/escalations/[decisionId]/outcome to record the human's decision. This feeds friction analytics and can automatically create RCA fault events if the agent was wrong to escalate.
receives fromVerifier FactoryDecomposition Designerfeeds intoRCA Engine
Module 08
Cost Benchmarks
Profiles each run for token spend, API call count, redundant calls, and cost per correct decision. Generates plain-language procurement recommendations at a specified volume.
1
View cost profiles
Each submitted run with LLM call traces automatically generates a cost profile: estimated USD cost, total tokens, API call count, redundant call count, efficiency percentile, and cost per correct decision.
2
Generate an efficiency report
Select a set of runs, set a projected monthly volume, click Generate efficiency report. The report compares agents on cost vs accuracy with annual cost projections. Download as PDF or JSON.
3
Redundancy detection
The engine automatically flags duplicate calls (same input/output within a session), verification loops (same verifier tool called more than twice), and hallucination recovery (rapid corrective calls).
receives fromVerifier Factory (run traces)feeds intoRegistry / Leaderboard
Module 09
Monitoring
Watches your agent system continuously. Configure thresholds and scenario suites; when a new run's scores fall below them — or a new fault type appears, or a regulatory violation is detected — Monitoring fires an alert.
1
Create a monitoring config
Click New config. Set: Scenario suite (which scenarios to watch), Feasibility threshold (minimum overall score 0–1), Completeness threshold, Fail on regulatory violation toggle, Auto-trigger RCA toggle, and notification channels.
2
Configure and test notification channels
Each config can notify via Slack webhook, email, or PagerDuty. Add the channel, save the config, then use Test incident delivery after the first monitoring run. Test events use the same redacted provider adapters as real incidents.
3
Enable / disable a config
Toggle the config active or inactive from the monitoring list. Inactive configs are preserved but do not trigger alerts.
4
Review alerts and delivery status
The alerts panel shows SCORE_REGRESSION, NEW_FAULT_TYPE, REGULATORY_VIOLATION, THRESHOLD_BREACH, and IMPROVEMENT. Recent delivery status records each provider attempt. Failed deliveries retry with bounded backoff without changing the evaluation result.
receives fromVerifier FactoryRCA EnginefiresSlack / Email / PagerDuty
Module 10
Investigate
Ask questions about agent behavior, RCA findings, and evaluation history in plain language. Every answer cites the real record it is grounded in; if Truvyx can't ground an answer, it says so instead of guessing.
1
Ask a question
Try: "why did run <run-id> fail", "is constraint blindness a recurring problem this month", or "which agent caused the most coordination breakdowns last week".
2
Read the grounded answer
Each answer links back to the specific run, constraint, or RCA report it's based on, and is labeled RULE_BASED, AI_ASSESSED, or MATHEMATICALLY_VERIFIED depending on how the answer was derived.
3
Track a recurring question
Click Track on a turn to have Truvyx watch for that same pattern automatically. Tracked patterns can be muted or deleted from the patterns panel.
4
Use it from Slack or your IDE
The same grounded-answer engine is available as a Slack command and through the MCP server (see MCP server), so you can investigate without opening the dashboard.
receives fromRCA EngineEvaluation Runssurfaces inDashboard / Slack / MCP
Configuration
Settings
Settings are at /settings. Six sub-pages: General, API Keys, Integrations, Members, Compliance, Exports, and Billing.
| Scope | Allows |
|---|
| runs:read | Read evaluation runs and results |
| runs:write | Submit new evaluation runs |
| scenarios:read | Read scenarios and scenario suites |
| scenarios:write | Create scenarios and publish suite versions |
| registry:read | Read the public registry |
| rca:read | Read RCA reports |
| monitoring:write | Trigger monitoring evaluation |
| telemetry:write | Send native or OTLP evidence |
| products:read / products:write | Read or manage product connections |
| chaos:write | Run chaos workflows |
🔒
Least privilege. For a CI pipeline that only submits runs, grant only runs:write. For a read-only dashboard, grant only runs:read. Never use an all-scopes key in automated pipelines.
Choose a 30, 90, or 365 day expiry where possible. Rotation immediately retires the previous secret and reveals its replacement once. Revoked, expired, and rotated identities remain in lifecycle history but cannot authenticate. Each successful use updates its last-used timestamp, and machine actions retain the stable service identity ID for audit attribution.
Slack
Enter your Slack incoming webhook URL. Once saved, select Slack as a notification channel in any Monitoring config.
Email (SMTP)
Enter SMTP host, port (typically 587), username, password, and From address. Works with Gmail, SendGrid, Postmark, etc.
PagerDuty
Enter your Events API routing key. Critical regulatory violations can be set to page on-call immediately when Fail on regulatory is enabled.
Compliance Mode
Toggle on to enforce stricter audit logging — all runs recorded with immutable verifier signatures and restricted-field enforcement is mandatory.
Owners control billing and organisation deletion. Admins manage lower-role members, API keys, audit controls, and credentials. Members create and operate evaluation resources. Viewers have read-only access. A user cannot grant or modify a role at or above their own, and invitees receive a seven-day email link.
For platform builders and investigators
How Truvyx joins execution evidence
Phase 2 modules record different kinds of evidence. Memory Evaluation records contamination, drift, hallucination, and conflict findings. Tool Fidelity records tool calls and correctness checks. Chaos Studio records injected faults and the agent's response. Truvyx keeps those native records as the source of truth, then exposes a read-time normalized event view so RCA, Investigate, Monitoring, and future pattern discovery can query one consistent shape.
Important: normalization does not rewrite, delete, or flatten the original module records. Missing semantics remain missing, and raw prompts, outputs, memory values, and tool payloads are not copied into the normalized metadata by default.
{
"traceId": "run_123",
"eventId": "p19:eval_456:call_7:call",
"eventType": "TOOL_CALL",
"startedAt": "2026-08-23T10:00:00.000Z",
"toolCallRef": "call_7",
"metadata": {
"sourceModule": "P19",
"toolName": "lookup_customer",
"parametersValid": false
}
}P17 · memory evidence
Memory findings become MEMORY_READ, MEMORY_WRITE, or DECISION events. The event keeps safe identifiers, scopes, finding types, and scores—not the memory content itself.
P19 · tool evidence
Each evaluated call produces a TOOL_CALL and a child TOOL_RESULT. Correctness, authorization, sequencing, latency, and the original call ID remain queryable without copying parameters or returned payloads.
P29 · chaos evidence
Each injected event becomes a tool or error event. Fault type, trigger state, classification, and source IDs are retained so resilience findings can be compared with ordinary execution evidence.
Browser and computer-use artifacts
Attach screenshots, Playwright traces, DOM or accessibility snapshots, console or network summaries, tool actions, and interaction logs to an evidence envelope. RCA previews byte-validated screenshots; other stored artifacts remain authenticated downloads. External video remains an HTTPS reference that Truvyx never fetches or embeds.
Artifact safety: stored binaries are limited to 10 MiB, screenshots to 5 MiB, text to 1 MiB, and metadata to 8 KiB. SVG and mismatched image bytes are rejected. Downloads are organisation-scoped, include a SHA-256 integrity digest, and expire with their evidence envelope.
Programmatic access
API & SDK
Supported programmatic workflows use the REST API and the TypeScript or Python SDK. Requests use Bearer authentication with your trk_-prefixed API key. The SDK Docs page at /docs/sdk provides live, copyable examples.
| Operation | Method | Endpoint | Scope |
|---|
| Submit a run | POST | /api/v1/evaluate | runs:write |
| Get run results | GET | /api/v1/runs/[id]/result | runs:read |
| Load a scenario | GET | /api/v1/scenarios/[id] | scenarios:read |
| Create a suite | POST | /api/v1/scenario-suites | scenarios:write |
| Read a suite | GET | /api/v1/scenario-suites/[id] | scenarios:read |
| Execute a suite | POST | /api/v1/scenario-suites/[id]/evaluate | runs:write |
| Get suite results | GET | /api/v1/scenario-suite-runs/[id] | runs:read |
| Trigger RCA | POST | /api/rca/analyze | rca:write |
| Get RCA report | GET | /api/rca/[runId] | rca:read |
| Record HITL outcome | POST | /api/escalations/[id]/outcome | runs:write |
| List registry entries | GET | /api/registry | registry:read |
| Trigger monitoring | POST | /api/monitoring/trigger | monitoring:write |
| Upload evidence | POST | /api/v1/evidence | telemetry:write |
| Attach evidence artifact | POST | /api/v1/evidence/artifacts | telemetry:write |
Authentication: Authorization: Bearer trk_your_key_here
Content type: All request bodies are application/json.
Coding agents & IDEs
MCP server
Truvyx also runs as a standard MCP (Model Context Protocol) server at mcp.truvyx.org. If your coding agent or IDE speaks MCP, it can search traces and investigate a run from inside your editor. Consequential calls such as scenario creation or verifier execution return a preview first and run only after the client repeats the unchanged request with its short-lived signed confirmation token.
1
Create a scoped API key
Go to Settings → API Keys (/settings/api-keys). Name it something like 'Claude Code – laptop' and create it. Copy the trk_ key immediately; it is shown only once. The Settings → API Keys page also has a ready-to-copy MCP config block once a key exists.
2
Get the server URL
The server lives at https://mcp.truvyx.org. Its manifest (tool list and schemas) is published at https://mcp.truvyx.org/.well-known/mcp with no authentication required, so a client can inspect what's available before you connect a key.
3
Add the server in your client
The exact UI differs per client, but every one of them wants the same two things: the URL above, and an Authorization: Bearer <your trk_ key> header (or an equivalent X-Truvyx-Key header). See the per-client commands below.
4
Verify the connection
Ask your coding agent to list its available MCP tools. You should see Truvyx tools such as investigate, search_traces, trigger_verifier, and create_scenario. If they don't appear, re-check the header name and that the key hasn't been revoked.
5
Start using it
Ask your agent things like 'use truvyx to investigate why run <id> failed' or 'trigger the verifier for scenario <id>'. Investigation runs immediately. For a consequential action, review the returned effect and let the client repeat the unchanged call with its signed confirmation token. A prose reply such as 'yes, proceed' is not authorization.
claude mcp add --transport http truvyx https://mcp.truvyx.org \
--header "Authorization: Bearer trk_your_key_here"
{
"mcpServers": {
"truvyx": {
"url": "https://mcp.truvyx.org",
"headers": {
"Authorization": "Bearer trk_your_key_here"
}
}
}
}🔑
Authenticate the same way as the REST API: a trk_-prefixed API key, passed as Authorization: Bearer or X-Truvyx-Key. Every tool call is scoped to that key's organisation. MCP is an open protocol, so any future MCP-compatible client works the same way, no Truvyx-specific integration needed.
🛡️
The public manifest labels every tool as read-only, low-risk write, or consequential and publishes its required scope. Truvyx records content-free audit entries for MCP mutations; it never stores the confirmation token or uses unrelated conversation text as confirmation.
Integration guide
Connecting your system
The recommended pattern keeps evaluation definitions in truvyx.yml, validates them on every pull request, and runs a selected scenario before deployment.
1
Commit your evaluation definitions
Add truvyx.yml to the repository with stable local scenario IDs, expected outcomes, constraints, and verifier configuration. Run truvyx-eval validate before committing.
2
Create a scoped API key
Go to Settings → API Keys. Create a key named after your pipeline and store it as TRUVYX_API_KEY in the CI secret manager. Synchronization requires scenarios:write; evaluation requires run-write access.
3
Review and synchronize changes
Run truvyx-eval diff truvyx.yml to review remote changes, then truvyx-eval push truvyx.yml. Truvyx records the remote mapping in .truvyx/scenarios.json.
4
Run the release gate
Capture your agent output as JSON, then run truvyx-eval run truvyx.yml --scenario <local-id> --input <file>. The command writes truvyx-report.json and links the canonical run result.
5
Retain evidence and monitor regressions
Upload truvyx-report.json even when the job fails. The gate blocks unless the canonical run passes, and always blocks failed HARD constraints or critical regulatory violations.
📋
Full SDK examples including TypeScript, Python, and cURL are on the SDK Docs page at /docs/sdk, pre-filled with your organisation's scenario IDs once you log in.
🧩
No CI pipeline? If you're driving Truvyx from a coding agent instead, use the
MCP server with the same scoped API-key authentication.
