From prompt to physical harm: mapping AI-enabled bioweapons risk pathways

01

Why map a pathway instead of debating a model in isolation?

A model does not create a real-world outcome alone. It sits inside a system of users, interfaces, retrieval, tools, agents, service accounts, institutions, and decisions. That system can broaden or narrow what a user can do. When the potential consequence is severe, a model-only discussion misses the very controls that determine whether a concerning output remains an abstract interaction or becomes part of a longer chain of action.

A high-level pathway map is not a playbook for harm. It is a safety exercise. It names categories of transition—request, access, interpretation, action capability, propagation, and recovery—so a team can ask where safeguards belong and what evidence would show that they worked. The map should deliberately omit operational biological detail. Its purpose is to reveal control gaps, not to teach someone how to exploit them.

The exercise also corrects a common mistake: treating a refusal as the only relevant outcome. A refusal may be appropriate, but the system still needs to ensure that data retrieval, tool calls, handoffs, and fallback pathways obey the same boundary. Conversely, a permitted high-level response can be appropriate when it is delivered within an authorised, observable, and constrained workflow.

02

The pathway has six questions, not six fixed steps

Different products have different architectures, so there is no universal sequence. Instead, use six questions that can be applied to any AI-enabled feature with life-sciences relevance. Each question identifies a place where the organisation can reduce uncertainty, narrow authority, or add a safe stop. The questions work for a simple assistant as well as a multi-agent system with connected services.

The answers should be concrete enough to test. ‘We have safety policies’ is not an answer to ‘Which connector can this workflow call in production?’ ‘A human is in the loop’ is not an answer to ‘At which condition does the system stop and which person is assigned?’ The map becomes useful when it produces named boundaries, expected evidence, and owners.

• Who is requesting assistance, and what organisation, role, purpose, and environment are verified?
• What information can the workflow reach, and how are source, classification, tenancy, and freshness bounded?
• What interpretation or planning authority does the AI have, and what uncertainty must cause it to pause?
• Which tools or downstream systems can act on its output, with what scope, expiry, and approval?
• How could a local error spread through agents, handoffs, integrations, or repeated use before detection?
• What can stop, investigate, reverse, and learn from a concerning event—and who owns each action?

03

Start with identity, purpose, and environment

The first control point is often before the model sees a request. The system should know whether the request is from an authenticated person or service, which organisation owns the session, which role applies, which product is being used, and whether the environment is development, staging, or production. These facts are not administrative decoration. They determine whether any later access or action is appropriate.

Purpose matters too. A broad account permission can be dangerous when it is reused for a narrow workflow. Bind access to the legitimate job the product is meant to support, and make the policy visible. When purpose, identity, or environment is missing or contradictory, the safe response may be to request clarification, limit the interaction to non-sensitive high-level material, or stop and escalate. The system should not infer a more permissive context because doing so is convenient.

This approach supports legitimate users as well as safety. Clear scope makes it easier to give authorised people dependable assistance, because the product knows which sources, tools, and outputs it may use. The alternative—an opaque, over-broad system that sometimes refuses and sometimes proceeds—is difficult for users and operators to trust.

04

Separate information access from action capability

A crucial design boundary is the difference between reading, reasoning, recommending, and acting. An agent may be permitted to explain a public concept but not to retrieve restricted records. It may be allowed to flag a case for a reviewer but not to trigger an external workflow. It may summarise evidence without gaining the authority to alter a downstream system. These distinctions should be explicit in architecture and evaluation, not inferred from the tone of a model response.

Connected tools deserve particular scrutiny because they turn an answer into an operational pathway. Apply least privilege: allow only the exact tool, target, parameter range, product, environment, and duration required. Require a current approval for a consequential action. Make an unavailable or ambiguous approval fail closed. A fallback provider or retry process should preserve the same rules rather than silently taking a broader route.

For AI-enabled bioweapons risk, this separation is a central containment principle. The harmful potential of a system is shaped not only by information but by the way information, access, and execution can be combined. A product team can reduce that risk without suppressing all useful scientific support by keeping those capabilities independently scoped and independently verified.

05

Test boundaries as a product behaviour, not a policy claim

Create a scenario for the actual workflow. Include the expected authorised case, an ambiguous case, a case with missing or expired authority, a cross-environment mismatch, and a dependency failure. State the expected safe outcome before running each case: complete within scope, request clarification, hold, block, or hand off to a qualified reviewer. Use synthetic and non-sensitive fixtures. The objective is to prove the control path, not to recreate a harmful request.

In Constraint Engine, encode the conditions that must hold. A product may require verified scope before a connector is invoked, an active review state before a sensitive workflow continues, or a complete trace before an evaluation can be passed. In Verifier Factory, check deterministic facts such as whether a forbidden tool call occurred or whether an approval was current. These are the building blocks of pre-deployment AI testing that a team can rerun after model, prompt, tool, or provider changes.

Where a condition requires expert judgement, use human review with a clear rubric and an appropriate privacy boundary. Do not ask an automated evaluator to certify a conclusion it cannot substantiate. The UK AI Security Institute notes that evaluations are not comprehensive declarations of safety; their value is in creating evidence that informs better decisions. The same humility should guide product release gates.

06

Look for propagation, not just the first mistake

A first mistake can be small: a missing field, an uncertain classification, an unavailable policy service, a handoff that loses an approval state. The question is whether the architecture lets that mistake travel. Does another agent treat an incomplete result as complete? Does a tool gateway accept a request without the original scope? Does a retry path lose the environment label? Does a dashboard show an unscored constraint as a green result? These are propagation questions.

Decomposition Designer can help make the dependencies visible. Model the agents, inputs, outputs, shared state, tools, approval points, retries, and fallback behaviour. Then test controlled failures in a non-production Chaos Studio experiment: a delayed handoff, a missing response, a provider outage, or an authentication failure. The expected result is often not graceful completion. It may be a safe stop with a clear, evidence-bearing alert.

If a run fails, use RCA to trace the evidence from symptom to candidate cause. Label hypotheses as hypotheses. Identify the smallest intervention that would have blocked the demonstrated pathway, then rerun the original scenario. Preserve it as a regression test. This is how a team prevents the same local gap from becoming a recurring systems problem.

07

Recovery is part of containment, not a postscript

A high-consequence control system needs a response plan before the alert. Define what a critical boundary violation means, who receives it, how quickly they respond, what can be paused, how evidence is preserved, and who can approve resumption. If the system cannot distinguish a product event from an evaluator or worker outage, it should say so; data unavailability must not look like a healthy result.

The incident record should connect the product, environment, scenario, constraint, run, evidence, owner, decision, and remediation. That allows an organisation to answer practical questions later: What happened? Which control was expected to stop it? Did the control run? Was the outcome observed or inferred? What changed afterwards? Did the correction hold under the original conditions?

The aim of this map is disciplined preparedness. It recognises that severe risks require more than a content filter, but it does not assume every system creates the same risk. By limiting authority, testing the transitions between layers, preserving honest evidence, and building a real escalation path, organisations can make AI-enabled systems more useful and more governable at the same time.