Evaluating Multi-Agent Systems in Healthcare: Navigating HIPAA and Clinical Constraints

Hallucination is broader than a fabricated medical fact. A system can cite a real guideline but apply it to the wrong age group, invent a missing laboratory value, mistake a planned medication for an active prescription, or silently treat an uncertain data match as confirmed. Each error changes the clinical meaning of an otherwise fluent response.

Multi-agent architecture adds propagation risk. An intake agent can normalize a value incorrectly, a retrieval agent can select an outdated protocol, and a recommendation agent can combine both errors into a confident plan. A final reviewer that sees only the polished answer cannot tell which assumption entered upstream.

Test facts and relationships separately. Fact checks confirm that medications, allergies, diagnoses, observations, and patient identity match authoritative records. Relationship checks verify that the system connected those facts correctly, including units, timing, status, provenance, and clinical relevance.

Severity must reflect the action boundary. An unsupported sentence in a clinician draft is serious, but an unsupported instruction sent directly to a patient or written into an order queue creates greater exposure. Scenario design should vary autonomy, reversibility, patient vulnerability, and time sensitivity rather than assigning one generic hallucination score.

Step-level traces make these distinctions observable. The step-level evaluation method shows how to attribute a final failure to the earliest unsupported claim instead of blaming whichever agent spoke last.

HIPAA testing begins with role and data flow, not with a prompt that says to protect privacy. Identify whether the organization and vendors are acting as covered entities or business associates, which systems create or handle electronic protected health information, and which permitted purpose authorizes each use or disclosure. Legal and privacy owners must determine the applicable rule set for the actual arrangement.

Turn each approved data path into an access predicate. A scheduling agent may need appointment type and availability but not a full clinical note. A care agent may need broader treatment context. HHS explains that the Privacy Rule generally requires reasonable steps to limit many uses, disclosures, and requests to the minimum necessary, while also identifying exceptions, including disclosures to or requests by a provider for treatment. A test should encode the relevant purpose and exception rather than apply one field limit to every workflow.

Security scenarios should cover confidentiality, integrity, and availability of electronic protected health information. HHS describes risk analysis as foundational and requires an accurate, thorough assessment of potential risks and vulnerabilities. Agent tests can contribute evidence by exercising authorization, isolation, tamper detection, failed-tool behavior, recovery, and vendor boundaries, but they do not replace the organization-wide Security Rule risk analysis.

Add adversarial cases. Attempt cross-patient retrieval, indirect prompt injection from a note, disclosure through logs, excessive context passed to a model provider, reuse of expired authorization, and a tool request from an agent whose role lacks permission. The expected result should name the blocked event and the retained evidence, not merely say that the model refused.

Preserve identifiers, versions, access decisions, tool calls, approvals, and disclosure destinations in an auditable record. Minimize sensitive payloads in evaluation artifacts and define retention and access rules for the artifacts themselves. A test system that copies unrestricted patient data into every trace can create the risk it is intended to measure.

Clinical constraints should come from approved protocols and local governance, with an owner, version, effective date, population, exclusions, required evidence, and escalation path. A guideline paragraph copied into a system prompt is not a reliable executable control.

Separate deterministic controls from judgment. Identity match, allergy conflicts, unit conversion, absolute thresholds, dose ceilings, required fields, and sequence rules can often be checked with code. Ambiguous symptoms, competing goals, and exceptions may require a clinician, but the system can still verify that the correct evidence and uncertainty reached that clinician.

Include missingness as a first-class state. A value can be present, absent, stale, pending, contradicted, or sourced from an unverified statement. Tests should punish invented completion and reward the system for requesting clarification or escalating when a required fact is unavailable.

Exercise temporal rules. Protocol eligibility can depend on age at the time of service, the interval since a procedure, the freshness of a laboratory result, or whether a medication was active when a symptom began. The temporal constraint guide explains how to represent these facts without flattening them into free text.

Measure workflow outcomes as well as recommendation correctness. A safe test can require no unauthorized order, complete provenance, clear uncertainty, escalation within the required window, and a final decision by the designated clinical role. These requirements prevent a high language-quality score from masking an unsafe process.

Consider a clinical nutrition assistant composed of intake, evidence retrieval, meal-planning, safety, and documentation agents. Its approved purpose is to prepare a clinician-facing draft, not to diagnose a condition or independently prescribe treatment.

Build the test fixture around a synthetic patient with chronic kidney disease, diabetes, a documented food allergy, a culturally important staple food, an uncertain supplement list, and laboratory results from different dates. Add a budget limit and limited food access so the plan must be safe and feasible rather than idealized.

Hard constraints include patient identity, allergy exclusion, approved data access, no fabricated laboratory values, clinician approval before patient delivery, and provenance for each patient-specific recommendation. Clinical thresholds must come from the organization's approved protocol and be evaluated against the correct units and effective time.

Quality dimensions include nutritional adequacy within the approved planning scope, adherence to patient preferences, affordability, explanation clarity, and a useful summary of unresolved questions. Those dimensions are scored only after every hard safety and privacy constraint passes.

The most informative failure may occur between agents. For example, the intake agent can correctly record an allergy while the planning agent loses it during context compression. The final safety agent might catch the error, but the run should still record the interface defect and create a regression case. Silent recovery is useful for the patient and insufficient for engineering assurance.

A release suite should cover routine cases, boundary values, protected subgroups, incomplete records, conflicting sources, unauthorized requests, malicious content, tool outages, delayed approvals, and attempted bypasses. Run repeated trials where model variability can change the path.

Report hard-constraint pass rate, unsupported clinical claim rate, identity and provenance accuracy, escalation recall, time to human review, privacy-control violations, and recurrence by root cause. Slice results by workflow, patient context, agent version, and autonomy level so averages do not hide concentrated risk.

Block release on any unresolved critical constraint failure. For lower-severity findings, assign an owner, due date, compensating control, and regression case. Production monitoring should watch the same events and predicates used before release, with incident procedures that protect patients and preserve evidence.

HIPAA compliance and clinical safety remain organizational responsibilities that depend on facts beyond a test suite. Agent evaluation makes important controls observable, repeatable, and reviewable. It does not certify a system by itself and this article is not legal or medical advice.