AI containment risks as global catastrophic risks: what organisations should prepare for now

01

Use the phrase carefully—or do not use it at all

Global catastrophic risk refers to outcomes so severe, widespread, and difficult to reverse that ordinary product-risk habits are not enough. It is not a synonym for a bad chatbot answer, an embarrassing automation error, or a quarterly operational incident. Treating every AI concern as catastrophic makes decision-making noisier and can obscure genuine priorities.

At the same time, organisations do not need certainty that a worst-case outcome will occur before they prepare for high-consequence failure. Where an AI system may increase the reach, speed, autonomy, or accessibility of a harmful capability, reasonable uncertainty is itself a governance fact. The appropriate response is proportionate: understand the pathway, limit avoidable exposure, test safeguards, and make sure a human organisation can intervene when signals are concerning.

That approach is consistent with the structure of the NIST AI Risk Management Framework: risk management is continuous across a system lifecycle and is organised around governing, mapping, measuring, and managing risks. It is not a checklist or a guarantee. It is a way to make decisions visible, evidence-led, and revisable as the product and its environment change.

02

What makes a containment failure high consequence?

The relevant question is not simply whether a model can generate concerning material. Context determines consequence. A low-authority assistant with no external tools, narrow access, strong review, and limited reach presents a different risk profile from a multi-agent workflow that can discover information, call services, coordinate across systems, and act at scale. Capability, access, autonomy, speed, and reversibility interact.

A high-consequence containment review therefore asks about the whole system. What information can it reach? Which tools can it invoke? Can it chain tasks without a human pause? Does it operate across tenants, environments, or external providers? Can an error spread before it is detected? Can the organisation identify exactly what happened and safely stop further action? This is a systems question, not only a model question.

Some risks remain uncertain or difficult to measure. That should lead to humility, not paralysis or false precision. Record what is known, what is inferred, and what has not been tested. A safety score is useful only when its scope, evidence, and blind spots are clear enough for someone else to challenge the conclusion.

• Scale: how many users, systems, or decisions can the workflow affect before intervention?
• Access: which data, tools, environments, and permissions are reachable in practice?
• Autonomy: which steps can happen without a timely human confirmation?
• Propagation: can one mistaken output trigger other agents, services, or operational workflows?
• Reversibility: can the action be stopped, corrected, or contained after it begins?
• Observability: can the team reconstruct the relevant evidence without exposing unnecessary sensitive data?

03

Preparedness begins with a bounded risk map

Teams often begin high-consequence discussions with a long list of abstract threats. A more useful starting point is one bounded product decision. Pick an AI system, its environment, its users, and one action that matters. Then map the intended benefit, the authority it needs, the safeguards already present, the foreseeable ways those safeguards could fail, and the human or technical control that should stop the path.

For example, a research-support system may be valuable because it helps authorised specialists navigate a large body of literature. A responsible risk map does not assume that a helpful summary is harmless in every context. It defines what content and tooling are in scope, how access is authenticated, which ambiguous or sensitive requests require a safe refusal or expert review, what telemetry is retained, and who can suspend the feature. It does not need to reproduce harmful material to test whether the boundary exists.

The World Health Organization’s guidance on responsible life-sciences use and dual-use risk offers a useful broader reminder: risk governance must account for benefits, potential misuse, and the institutions that make decisions around them. For AI teams, that means connecting product controls to real owners, escalation routes, and review practices rather than treating safety as a static policy document.

04

Convert preparedness into testable release conditions

A risk map becomes operational only when it creates release conditions. In Scenario Studio, define the task narrowly enough to evaluate. Name the product and environment. Write the expected safe behaviour for an ordinary request, an ambiguous request, an out-of-scope request, and a dependency failure. Add the roles involved and the evidence that must be present before an action can proceed.

In Constraint Engine, express non-negotiable boundaries as hard constraints. Examples at the system level include: the action must remain in the approved environment; a required review state must be current; an unauthorised tool path must not be used; a missing provenance field must cause a hold or escalation. These are examples of control patterns, not universal rules. Each product needs conditions that match its own risk and authority.

Then test the controls with a mix of deterministic verifiers and carefully scoped qualitative review. A deterministic verifier can establish whether a required approval or scope check occurred. Human reviewers or calibrated evaluators can inspect whether an escalation was clear and appropriate. If a condition cannot be assessed because the trace is absent or a verifier is unavailable, record it as not scored. Claiming a pass without evidence is exactly the kind of visibility gap a containment programme is meant to prevent.

05

Independent challenge is part of the control, not an optional ceremony

A team closest to a product has valuable knowledge, but it also has incentives and assumptions that can narrow its view. High-consequence preparation benefits from structured challenge: a domain expert who understands the real-world context, a security or safety reviewer who can inspect boundaries, and an operational owner who knows whether the organisation can respond under pressure. The goal is not to create an endless approval queue. It is to make consequential assumptions discussable before they become production defaults.

Keep the review grounded in artefacts. Review a scenario, the constraint set, the verifier results, the evidence requirements, the escalation policy, and the monitoring configuration. Ask what would make the team pause deployment, what signals would trigger review, and what it would take to reverse a decision. The NIST framework explicitly emphasises documented roles, ongoing monitoring, testing before deployment and during operation, and independent review as ways to improve risk measurement and governance.

For organisations using AI agent evaluation tools, this is a practical form of Machine Trust. The system earns trust because an interested reviewer can see what was tested, what was not tested, which boundaries held, who owns the response, and where uncertainty remains. It does not earn trust by presenting a single universal score.

06

Build an escalation system before a difficult signal arrives

Containment depends on what happens after a concerning event as much as before it. Decide in advance which signals block a release, which create a Quality Issue, which automatically begin an RCA investigation, and which require an executive or domain-owner review. Define a safe stop, a communication route, a record of the decision, and a condition for resuming service. A plan that says only ‘notify the team’ has not yet specified a response.

Monitoring should be able to distinguish a product failure from an evaluator outage. If a worker cannot process a run, if a provider is unavailable, or if evidence cannot be collected, the dashboard should state that explicitly. Operational uncertainty is itself a condition to manage. It should not disappear into a green status or an empty trend line.

After any meaningful incident or near miss, preserve the case as a regression test. Use RCA to label observed facts separately from causal hypotheses. Make the smallest repair that addresses the demonstrated gap, run the original and corrected cases, and document the result. This is slower than declaring the system safe after one successful test. It is also how an organisation builds credible assurance over time.

07

What preparedness looks like next week

The most useful next step is modest and concrete. Choose one workflow with meaningful authority. Create a product profile and environment. Map its access, tools, handoffs, and human stops. Write a scenario that includes an ordinary case and a case where critical evidence is missing. Make the expected safe outcome explicit. Run it, inspect the evidence, assign an owner, and preserve it as a regression case.

Repeat that process as the system gains tools, autonomy, users, or integration pathways. Bring in the people who understand the affected domain. Reduce unnecessary access. Keep review and monitoring proportionate to the plausible consequence. And be honest when coverage is incomplete. In high-consequence settings, an unmeasured risk is not solved by a reassuring label; it is a reason to decide whether more testing, stronger controls, or a narrower deployment is needed.

That is the practical meaning of preparing for AI global catastrophic risk: neither panic nor complacency, but disciplined restraint backed by evidence. The organisation learns to recognise the boundary of its knowledge, contain what it can, and escalate what it cannot safely resolve alone.