Use Truvyx from your first product connection to continuous quality control.

Before you write a test, create the AI system that will be tested. A product is not a billing item and it is not your entire company. It is one AI capability with a clear operating context, such as a claims-triage assistant, a support copilot, or a multi-agent research workflow.

The environment matters just as much as the name. Development, staging, and production can legitimately produce different behaviour. Recording the environment prevents a staging experiment from being read later as production evidence. Use the deployment identifier or Git SHA whenever you have one, so a future failure can be tied to the version that caused it.

Select the product you just created, then choose the pathway that matches where your AI system already lives. REST/cURL is the broadest option. The TypeScript and Python SDKs suit application code and notebooks. GitHub Actions, GitLab CI, and generic CLI support release gates. MCP is for coding agents. Custom telemetry is for observed production behaviour. Manual submission is useful for the first example or a debugging session.

Choosing a pathway records intent and opens a setup panel. It does not prove the connection works. Complete the setup in your own system, send a safe test request, then use Mark setup verified only when you have received and inspected the expected response. Telemetry is unscored until you explicitly map it to a scenario or monitoring configuration.

At the beginning, an empty or sparse Overview is normal. It becomes useful after you create scenarios, submit runs, send approved telemetry, configure monitoring, and create RCA reports. Its tiles deliberately count different things: an evaluation run is not a telemetry event, and an open alert is not an RCA diagnosis.

Use the trend selector to ask one question at a time. A falling pass rate suggests a quality regression. A rising telemetry count with no evaluations may mean events are arriving but have not been mapped to a monitoring rule. The recent-activity list links back to the source record so you can inspect evidence instead of acting on a summary alone.

A good first scenario is narrow. Instead of ‘evaluate our support bot,’ use a decision with a consequence: ‘decide whether a claims case can proceed, must be escalated, or needs more evidence.’ Name the agents involved, the information they may use, what they should produce, and what must never happen.

Choose From Template when one of the supplied domains is close to your workflow. Choose From Natural Language when you need to describe a specific operation in ordinary language. The natural-language provider creates a draft specification; it does not become policy until you read and save it. If the provider is unavailable, use the template path or write the fields manually rather than assuming the scenario was created.

Use HARD for a rule that must not be violated. A hard rule should say what must be observed and what should happen when the evidence is absent. Use SOFT for an optimisation or preference that can influence the score without making the whole run unacceptable. Do not use a soft score to hide a regulatory or safety requirement.

Spread the first scenario across categories. A real multi-agent workflow often needs a safety or regulatory rule, a data-governance rule, a temporal or dependency rule, and an optimisation preference. The category is useful later: it lets you see whether a regression is concentrated in timing, access boundaries, handoffs, or cost.

From a scenario, generate or open its verifier. A good verifier makes checks repeatable: it records the scenario version, verifier version or checksum, the structured output it examined, and evidence for each result. A score without a per-constraint explanation is not enough for a consequential decision.

For the first run, manual submission is useful. Paste a real or representative structured output, select the scenario and product context, name the agent system, and submit. For normal engineering work, send the same payload from your SDK, REST integration, or CI workflow. Treat PENDING or RUNNING as non-terminal states; do not read them as a pass.

Use this module when a final answer alone cannot explain quality. A multi-agent system can fail even when each individual model response appears reasonable: an agent can receive stale context, a dependency can be missing, two agents can own the same task, or a retry can loop without a bound.

Start from the scenario, then represent the workflow honestly. A node should have an owner and a purpose. An edge should describe the input or handoff, including any approval point. The diagram is not decoration; it is the design that Scenario Studio, Constraint Engine, Verifier Factory, and RCA use to make failures easier to locate.

Registry prevents a team from repeatedly rebuilding the same evaluation assets. It is the place to find something that has an owner, version, status, tags, dependencies, approval state, and evaluation history. It should not be used as a dumping ground for drafts that have never been reviewed.

Before publishing, make sure the item has a clear purpose, stable version, appropriate visibility, and an associated evaluation or quality check. Use the resulting registry record when another team needs a trusted starting point, not merely a copy of some text from a previous project.

Open RCA from a failed run. Truvyx reads the evaluation evidence, trace references, workflow context, tool and memory events where available, and related patterns. It separates observed symptoms from a causal hypothesis. That distinction is essential: an AI-generated explanation is not proof simply because it sounds plausible.

A useful RCA report gives you a causal chain, provenance or confidence, linked evidence, counterfactual interventions, an owner, and a way to verify the repair. If the evidence is incomplete, the report must say so. The correct next action may be to collect a trace or rerun a controlled case rather than change production behaviour.

Ask a focused operational question, such as ‘Why did this run fail?’, ‘Has this safety violation happened before?’, or ‘Did failures change after the staging deployment?’. Investigate should return a direct answer together with linked source records, the filters or time range used, evidence excerpts where safe, and a provenance or confidence statement.

It is acceptable for the answer to be ‘there is not enough data.’ A trustworthy investigation does not invent a pattern from one run, expose raw customer content, or turn a missing trace into a causal conclusion. Use the linked records to verify a surprising answer before changing a system.

Chaos experiments should be isolated from production by default. Begin with a staging product, a known scenario, an explicit abort condition, and a narrow fault. The point is not to make the system fail dramatically. The point is to see whether it fails safely, records the right evidence, respects boundaries, and recovers as designed.

Useful first faults include a bounded latency injection, a timeout, a missing tool response, an authentication error, payload corruption, stale memory, a delayed handoff, or provider unavailability. Decide what acceptable behaviour looks like before you run the experiment: retry once, use a fallback, escalate to a human, block the result, or stop the workflow.

Human-in-the-loop is not a checkbox. Record why the agent escalated, how long the review took, whether the reviewer accepted, rejected, or edited the result, and whether the reviewer agreed with the agent and verifier. This turns an invisible manual safety net into a measurable part of the system.

Use outcomes by scenario, model, agent, product, and environment. A high override rate can mean the model is weak, the constraints are incomplete, the escalation threshold is too low, or reviewers are correcting a workflow defect. Follow the record into the original run and RCA report before drawing conclusions.

A low-cost run is not automatically efficient. If it skipped a verifier, omitted an evidence source, or bypassed human review, it is not comparable with a fully evaluated run. Use the same scenario, version, constraints, and output requirements when comparing a model, routing strategy, or agent decomposition.

The useful measures are cost per correct decision, tokens per satisfied critical constraint, cost and latency by agent, redundant calls, p50 and p90 cost, and projected spend at volume. Capture safe token and latency metadata through your integration so the benchmark has enough information without receiving raw prompts or secrets.

Enable monitoring only after you understand at least one passing and one failing evaluation. Choose a baseline run or approved scenario version, decide which score or constraint should trigger action, and make the notification route explicit. A monitoring configuration is an operational promise, so it needs an owner and an environment.

An alert should say what changed, when it changed, which product and scenario detected it, which constraint failed, whether evidence is complete, whether RCA is available, and what to do next. Monitoring is not a substitute for a run detail page; it is the service that tells you when to open one.

Create an issue when a failure is material, recurring, or needs work across teams. Link it to the representative run, relevant constraints, RCA report, product, environment, and monitoring configuration. This preserves the path from a dashboard symptom to the repair that actually closed it.

Do not resolve an issue because a single rerun happened to pass. A sound closure records the change, owner, verification run, remaining uncertainty, and whether a regression suite or monitoring rule now protects against recurrence.

Use the versioned REST endpoint as the common denominator: submit with POST /api/v1/evaluate, then retrieve a result with GET /api/v1/runs/[runId]/result. The request uses camelCase fields such as scenarioId, agentSystemName, and agentOutput. The response gives a run ID; your pipeline should poll until a terminal result and then decide whether the threshold or critical-violation policy blocks the build.

Use TypeScript or Python only where the published client package and documentation match the current API. Until a Python package is officially published to PyPI, follow the repository or REST instructions shown in SDK Docs rather than advertising an unavailable pip package. GitHub Actions, GitLab CI, and generic CLI integrations should store trk_ keys as secrets and upload a result report as an artifact.

MCP is best when a developer works from Claude Code, Cursor, Windsurf, Cline, Zed, Goose, or another compatible client. Create an API key with the smallest required scopes, add https://mcp.truvyx.org to the client, and supply the key in Authorization: Bearer or X-Truvyx-Key. The public manifest describes available tools; authenticated calls are limited to the key’s organisation.

MCP should not silently ingest customer data. A visible tool call should create or find a scenario, trigger a verifier, inspect a run, search traces, or start an investigation. The resulting run remains linked to the dashboard so the same evidence is available to people who are not in the developer’s editor.

Complete Settings before a production rollout, not after an alert arrives. Set the organisation details, invite only the teammates who need access, choose roles deliberately, and create named keys with narrow scopes. An owner should know where keys are used and should be able to revoke them without asking an engineer to search through code.

Connect Slack, email, or PagerDuty only after deciding who owns which alert. Configure model providers for Truvyx’s own assisted features separately from customer-product connections. Review compliance and restricted fields, data retention, exports, and audit records with the people responsible for risk and privacy. Never put raw prompts, outputs, or secrets into general-purpose analytics simply to make a dashboard more complete.