Feasibility, completeness and optimality: a better AI scorecard

01

Why one score is not enough

Suppose an AI agent produces a confident answer to a scheduling request. It sounds helpful, but it booked a time outside the customer’s availability and forgot to mention a required preparation step. A single pass rate might call this a success. A useful scorecard would not.

AI systems combine several jobs: obeying constraints, covering the requested work, and making a sensible choice among valid options. Separating those dimensions makes failures easier to explain and fixes easier to verify.

02

Feasibility asks: could the action be allowed?

Feasibility is the foundation. It measures whether the agent stayed inside the rules of the scenario: permissions, deadlines, budgets, information boundaries, safety rules, and regulatory constraints. A response that is beautifully written but impossible or prohibited is not feasible.

For example, an AI work planner may suggest a resource that is already booked. The plan can be complete, but the action is not feasible. Constraint Engine and verifier checks are particularly useful for this layer.

• Hard constraints: conditions that must never be violated
• Soft constraints: preferences that influence ranking
• Regulatory or safety constraints: failures that may override the overall score

03

Completeness asks: did the system finish the job?

Completeness measures coverage. Did the agent answer every part of the request? Did it return required fields? Did it perform the handoff, cite the evidence, or explain the exception? This catches a common weakness in AI agents: they do one part impressively and quietly omit the rest.

A beginner-friendly way to write a completeness check is to make a checklist from the user’s request. If the request has three required outcomes, the evaluation should show three pieces of evidence. Missing one should be visible instead of disappearing into a fluent paragraph.

const required = ["decision", "reason", "nextStep"]
const missing = required.filter((field) => !output[field])
const completeness = 1 - missing.length / required.length

04

Optimality asks: was this a good valid choice?

Optimality is different from feasibility. Many actions can be allowed, but some cost more, take longer, or create unnecessary risk. Optimality compares a valid solution with the objective of the scenario. In a multi agent system, this may include fewer tool calls, a shorter path, or a better balance of speed and certainty.

Do not use optimality to punish a safe fallback. If the system escalates because it lacks evidence, that may be the correct choice even if it is slower. Define the objective before scoring it.

05

Combine the metrics without losing the story

A weighted score can provide a useful summary, but keep the components and violations beside it. Truvyx documents an overall weighting of 0.4 feasibility, 0.35 completeness, and 0.25 optimality. That ordering reflects a sensible principle: a fast, elegant violation is still a failure.

Use the scorecard to ask better questions: did the model change reduce feasibility, did a prompt change create completeness gaps, or did a new tool path increase cost? Those questions lead to repairs. A number alone does not.

06

Walk through one result like a teacher

Imagine an agent receives a request to reschedule a delivery. It chooses a valid slot, but forgets to confirm the customer’s address and makes four unnecessary tool calls. Its feasibility may be high because the chosen slot is allowed. Its completeness is lower because a required confirmation is missing. Its optimality is lower because the path was wasteful.

This example shows why a scorecard helps a beginner read results. Ask one question per dimension, write down the evidence, and only then look at the combined score. The order prevents the summary number from hiding the actual story.

07

Set thresholds from real behaviour

Run a small baseline of representative cases before choosing a release threshold. Include easy cases, edge cases, and cases where a safe escalation is the right answer. If the baseline varies widely, investigate the scenario or rubric before tightening the gate.

A threshold is a decision boundary, not a universal definition of quality. A regulated workflow may require a perfect hard-constraint result even when its average score is high. A creative assistant may care more about completeness and usefulness. Write down why each threshold exists so the next person can maintain it.