Safe State And Graceful Degradation
Objective
Define and implement a minimum-risk condition for each AI-controlled physical system, reached automatically when AI confidence falls below threshold, anomaly is detected, or human override is activated.
Control / requirement
For each AI-controlled system, document: safe state definition (autonomous vehicle: controlled stop; surgical robot: tool withdrawal; industrial arm: immediate stop and hold); transition time to safe state (must be within stopping distance/reaction time for physical context); trigger conditions for safe state entry; recovery procedure. Implement degraded mode ladder: Full AI control → AI-assisted human control → Manual-only → Safe state.
Business impact
Absence of a defined safe state means any AI failure — model error, adversarial attack, or communication loss — can result in uncontrolled physical system behaviour. Uncontrolled physical behaviour in industrial or automotive contexts represents potential for catastrophic irreversible harm and unlimited product liability.
Validation approach
Test ID: D9-CTL-02-VTS-001 | Test Type: Hybrid (Simulation + Manual) | Harness: pytest tests/d9_physical_ai/test_safe_state_degradation.py -v # oda3-gaissf-vts | Pass Criteria: safe_state_reached_within_spec = 100% of trigger condition tests; safe_state_physically_safe = True (verified by independent assessor). Test: inject each trigger condition; measure transition time and safe state accuracy. Independent assessor verifies safe state is physically safe for operational environment.
Expected evidence
Not separately specified in the available source.
Mapping and source
Not separately specified in the available source.
Implementation guidance
Translate the requirement into system-specific procedures, responsible roles, technical configurations and review conditions. Retain evidence showing both design and operating performance. Where the source provides no separate implementation instruction, do not infer that a single technical mechanism is sufficient.
Assessment considerations
- Confirm scope and applicability.
- Inspect control design and responsible ownership.
- Test representative operation and adverse conditions where appropriate.
- Evaluate evidence provenance, completeness and contradictory evidence.
- Record limitations and notably absent outcomes.