Catastrophic Risk Is Not Just a Lab Problem
Catastrophic AI risk is often discussed in connection with the companies building the most capable frontier models. That makes sense. Frontier developers can evaluate model capabilities, design model-level safeguards, and investigate failure modes before those models reach downstream users.
But legal scope and risk relevance are different questions. An organization can be outside a frontier-AI law and still depend on a frontier model. Once that model is connected to organizational data, tools, credentials, users, and automated workflows, the deployment has a risk context the model developer does not fully control.
Find the Risk Ceiling
Catastrophic-risk thinking offers practitioners a useful discipline: begin with severity. Ask what the most severe reasonably foreseeable harm is that the AI system could cause or materially contribute to. Then work backward to understand what capabilities, conditions, failures, and control weaknesses would have to exist for that outcome to occur.
This is not a prediction. A maximum-harm scenario may be unlikely. The purpose is to avoid allowing a low estimated likelihood to hide a consequence that deserves explicit consideration. NIST's AI Risk Management Framework similarly treats risk in terms of both the likelihood and magnitude of consequences and recognizes impacts to individuals, groups, communities, organizations, and society.
For many systems, the answer will not be catastrophic. That is useful information too. A customer-service assistant with narrow permissions has a different risk ceiling from an autonomous agent with privileged production access. The exercise helps establish that difference rather than assuming all AI systems deserve the same controls.
Risk Travels Downstream
Upstream safety work can become risk intelligence for downstream organizations. If a model provider identifies a consequential capability, limitation, or failure mode, a downstream risk manager can ask whether the organization's deployment creates a credible pathway from that capability to harm.
| Upstream | Deployment | Potential consequence |
|---|---|---|
| Model capabilities and safety evaluations | Access, autonomy, data, tools, scale, and business process | Real-world effects on people and systems |
| Provider safeguards and assumptions | Customer controls and configuration | Residual risk after both layers operate |
This does not mean the downstream organization automatically shares the provider's legal obligations. It means the organization's risk assessment should account for dependencies it does not own and conditions it does control.
Questions to Take With You
- What is the most severe reasonably foreseeable harm this system could cause or materially contribute to?
- Who could experience that harm, including people outside the organization?
- What model capability would be necessary for the scenario?
- What does our deployment add: access, autonomy, sensitive data, tools, scale, or authority?
- Which provider assumptions are we relying on?
- What prevents the scenario, and what happens if that control fails?
Part 2 looks at why California and Illinois put catastrophic-risk duties at the frontier-developer level. Part 3 then borrows from safety engineering to turn the backward-reasoning idea into a practical analysis.