RAI Black Box

Summary: The purpose of embedding an RAI black box in an AI system is to investigate why and how an AI system caused an accident or a near miss.

Type of pattern: Product pattern

Type of objective: Trust

Target users: Architects, developers

Impacted stakeholders: RAI governors, AI users, AI consumers

Relevant AI principles: Human, societal and environmental wellbeing, human-centred values, fairness, privacy protection and security, reliability and safety, transparency and explainability, contestability, accountability

Mapping to AI regulations/standards: EU AI Act, ISO/IEC 42001:2023 Standard.

Context: AI systems may randomly fail or malfunction. If the cause of the problem cannot be tracked, these failures may reappear and cause more serious damage to humans, society, and the environment.

Problem: How can we keep track of AI systems’ behaviors and decisions and explain them when accidents occur?

Solution: The black box was introduced initially for aircraft several decades ago to record critical flight data. The intention of adding a black box to aircrafts is to collect evidence of the actions of the system and the surrounding context information for analysis after near misses and failures. The near misses and failures are specific to the use cases. Although the primary usage of a black box is accident investigation, black boxes are useful for other purposes. Data collection and the analysis could support improvement of the system. The purpose of embedding an RAI black box in an AI system is to investigate why and how an AI system caused an accident or a near miss. As shown in Figure 1, the RAI black box continuously records sensor data, internal status data, decisions, behaviors (both system and operator), and effects [1].

 

For example, an RAI black box could be built into the automated driving system to record the behaviors of the system and driver and their effects [2]. All this data needs to be kept as evidence with the timestamp and location data. Designing the RAI black box is challenging because the ethical metrics need to be identified for data collection. Also, design decisions need to be made on what data should be recorded and where the data should be stored (e.g., using a blockchain-based immutable log or a cloud-based data storage) [1].

Fig.1 RAI black box

Benefits:

  • Accountability: An RAI black box is critical to the investigation of why and how the AI system caused accidents.
  • Traceability: An RAI black box continuously records sensor data, internal status data, decisions, behaviors (both system and operator), and effects.

Drawbacks:

  • Privacy: Sensitive data might be collected by the RAI black box.

Related patterns:

  • Global view auditor: Global-view auditor might work with RAI black box when the data collected by the RAI black box is analyzed.

Known uses:

  • RoBoTIPS aims to develop an ethical black box for social robots, to enable the explainability of their behavior.
  • Falco et al. propose to drive widespread assurance of highly automated systems via independent audit using “black box”.
  • Falco et al. propose a “black box” audit trail for automotive software and data provenance based on distributed hash tables (DHT).

References

[1] Falco, G. and J.E. Siegel. A DistributedBlack Box’Audit Trail Design Specification for Connected and Automated Vehicle Data and Software Assurance. arXiv preprint arXiv:2002.02780, 2020.

[2] Winfield, A.F. and M. Jirotka. The case for an ethical black box. in Annual Conference Towards Autonomous Robotic Systems. 2017. Springer.