Fault Tree Analysis (FTA)

Summary: FTA is a top-down risk assessment method that is used to analyze the root causes of potential RAI risks associated with the development and use of an AI system.

Type of pattern: Governance pattern

Type of objective: Trustworthiness

Target users: Project managers

Impacted users: Development teams

Lifecycle stages: Requirements engineering, testing, operation

Relevant AI ethics principles: Human, societal and environmental wellbeing, human-centered values, fairness, privacy protection and security, reliability and safety, transparency and explainability, contestability, accountability

Mapping to AI regulations/standards: EU AI Act, ISO/IEC 42001:2023 Standard.

Context: Compared to traditional software systems, AI systems have a higher degree of risk due to their multi-level artifact dependency, data-dependent behaviors, and potentially opaque decision-making process. Undesired system behaviors or decisions could lead to serious consequences and even cause loss of human lives. An RAI risk assessment is a critical activity to ensure AI systems are developed and used in a trustworthy and responsible way.

Problem: How can we determine the cause of potential ethical failures and anticipate the RAI risks?

Solution: FTA is a method that describes how system-level ethical failures are led by small ethical failure events through an analytical graph, known as a fault tree [1]. The development team can easily understand how ethical failures propagate throughout the AI system by using FTA. FTA can be performed during the design or operation stage to anticipate the potential RAI risks and to recommend mitigation actions.

Benefits:

• Reduced RAI risk: FTA helps analyze the RAI risks related to AI system artifacts and identify the context conditions under which an AI system is unethical.
• Prioritized risk: FTA prioritizes ethical issues that contribute to an RAI risk.

Drawbacks:

• Lack of scalability: For larger systems, FTA can be complex and may involve many ethical events and gates.
• Inefficiency: Each FTA graph is designed to only examine one top event.
• Missing factors: Time can hardly be captured in FTA.

Related patterns:

• RAI Risk Assessment: FTA is a top-down method for assessing RAI risks.
• Failure Mode and Effects Analysis (FMEA): FMEA adopts a bottom-up approach investigating what may cause ethical failures while FTA is a top-down approach assessing possible ethical failures. FTA examines the relationship between different ethical events while FMEA does not.

Known uses:

• FTA was firstly introduced by Bell Laboratories in 1962 to assess the safety of a missile launch control system.
• Boeing started using FTA to design civil aircrafts from 1966.
• FTA was included in U.S. Army Materiel Command’s Engineering Design Handbook on Design for Reliability.

References:

[1] Ebert, C. and M. Weyrich, Validation of Autonomous Systems. IEEE Software, 2019. 36(5): p. 15-23.