Failure mode and effects analysis (FMEA)
Summary: FMEA is a bottom-up risk assessment method that is used to identify and analyse RAI risks.
Type of pattern: Governance pattern
Type of objective: Trustworthiness
Target users: Project managers
Impacted users: Development teams
Lifecycle stages: Requirements engineering, testing, operation
Relevant AI ethics principles: Human, societal and environmental wellbeing, human-centered values, fairness, privacy protection and security, reliability and safety, transparency and explainability, contestability, accountability
Context: Ethical issues in AI systems are often identified through extensive simulation and testing during the later stages of development. However, this can lead to significant delays in project timelines and an increase in development cost. By identifying and address ethical issues early on in the development process, development teams can mitigate the ethical issues and avoid costly delays.
Problem: How can we ensure the ethical quality at the beginning of the development process?
Solution: FMEA is a systematic and qualitative RAI risk assessment method to identify and evaluate potential RAI risks [1]. This bottom-up approach allows the development team to gain a comprehensive understanding of the potential failure modes, their causes, and the impacts of the failures on the systems and their users. FMEA can provide a clear view on the mitigation actions to reduce occurrence frequency and impact and increase detection probability.
Benefits:
- Improved ethical quality: FMEA ensures that ethical failures never happy in the first place by analyzing all potential ethical failures.
- Ease of use: FMEA is relatively easy to use in practice.
- Early identification: FMEA provides early identification of ethical failures and helps to avoid delays to schedules.
Drawbacks:
- Limited by expertise: FMEA replies on experts to apply their professional knowledge and experience to the RAI risk assessment process. Thus, the quality of the analysis is limited by the expertise of the team performing the analysis.
- Missing failures: FMEA is better suited for bottom-up analysis and not able to detect complex system-level ethical failures that require a holistic perspective.
Related patterns:
- RAI Risk Assessment: FMEA is a method of RAI risk assessment focusing on the development process and product design.
- Fault Tree Analysis (FTA): FTA assesses each of the possible ethical failures, while FMEA focuses on root causes that may lead to failures.
Known uses:
- FMEA was originally proposed in US Armed Forces Military Procedures document MIL-P-1629 in 1949 and revised in MIL-STD-1629A in 1980.
- Ford Motor Company firstly introduced FMEA to the automotive industry for assessing safety risk since mid 1970s.
- FMEA has been extended and adopted by Toyota’s Design Review Based on Failure Modes (DRBFM) for assessing potential risk and reliability for Automotive and Non-Automotive applications.
References:
[1] Ebert, C. and M. Weyrich, Validation of Autonomous Systems. IEEE Software, 2019. 36(5): p. 15-23.