Borrow From Safety Engineering
Working backward from an undesirable outcome is not new. Safety and reliability disciplines use related techniques to understand how serious failures can occur.
| Technique | Plain-language idea | Useful AI question |
|---|---|---|
| Fault tree analysis | Start with an undesirable top event and identify combinations of failures that could cause it. | What would have to fail for this harm to occur? |
| Bow-tie analysis | Map causes and preventive barriers before an event, then consequences and mitigating barriers after it. | What prevents the event, detects it, limits it, and supports recovery? |
| Backward reasoning / backcasting | Start with an outcome and work backward through necessary conditions. | What capabilities and deployment conditions make this scenario credible? |
AuditDIFF uses these as influences, not as interchangeable standards. The practical goal is to avoid starting an AI assessment with whatever controls the organization already happens to have.
The Risk Chain
Begin by defining the most severe reasonably foreseeable harm. Then ask what scenario could produce it. Identify the AI capability that scenario requires, the deployment conditions that expose or amplify the capability, and the safeguards that would have to fail or be absent.
NIST's AI RMF supports the underlying discipline even though it does not prescribe this exact AuditDIFF sequence. The framework calls for organizations to map context and impacts, measure risks, and manage prioritized risks while considering both likelihood and magnitude.
Work Back to Controls
Once the pathway is visible, controls have a job. Each control should interrupt, detect, limit, or help recover from part of the pathway.
This view also exposes single points of failure. If every severe scenario depends on one human approval step, the auditor should ask how that approval works, whether it can be bypassed, what evidence it leaves, and what happens when the reviewer makes a mistake.
A Simple Example
Consider an AI agent allowed to make changes to cloud infrastructure. A severe scenario might involve the agent taking an unauthorized action that disrupts a critical production service.
| Layer | Example |
|---|---|
| Maximum credible harm | Critical service becomes unavailable for an extended period and people who depend on it are affected. |
| Capability | The agent can generate and execute infrastructure changes. |
| Deployment condition | The agent has privileged production credentials. |
| Preventive control | High-impact changes require independent approval and constrained permissions. |
| Detection | Changes and anomalous actions are logged and monitored. |
| Response | Access can be revoked, changes rolled back, and the agent disabled. |
| Evidence | Permission configuration, approval records, logs, alerts, rollback tests, and incident exercises. |
The exercise does not prove catastrophe is likely. It makes the pathway testable. Part 4 adds the upstream model provider to that pathway.