RAI Assessment for Test Cases

Summary: All the test cases for RAI acceptance testing should pass RAI assessment.

Type of pattern: Process pattern

Type of objective: Trustworthiness

Target users: Testers

Impacted stakeholders: Developers, data scientists

Lifecycle stages: Testing

Relevant AI ethics principles: Human, societal and environmental wellbeing, human-centered values, fairness, privacy protection and security, reliability and safety, transparency and explainability, contestability, accountability

Mapping to AI regulations/standards: EU AI Act.

Context: Ensuring the ethical quality assurance for AI systems relies heavily on RAI acceptance testing, which aims to identify and solve ethical concerns with the AI system. A set of test cases with expected results should be maintained to detect possible ethical failures in various extreme situations. However, it is also important to note that the test cases themselves may contain ethical issues, such as fairness or privacy issues in the test data.

Problem: How can we ensure the ethical quality of test cases?

Solution: Creating high-quality test cases is an integral part of RAI acceptance testing. A test case usually includes an ID, description, preconditions, test steps, test data, expected results, actual results, status, creator name, creation date, executor name, and execution date. All the test cases for verification and validation must pass an RAI assessment, which includes evaluating the RAI metrics of the test steps and test data. Creation and execution information is essential to track the accountability of ethical issues with test cases. The assessment process can be integrated into the design of tools used to generate test cases.

Benefits: 

  • Ethical quality: RAI assessment can help improve the overall quality of the test case by identifying and address any ethical issues in the AI systems.
  • Adherence to ethical principles: By conducting RAI assessment, developers can ensure that the test cases align with ethical principles.

Drawbacks:

  • Limited scalability: New test cases need to be continually added and assessed when new ethical requirement emerge or the operational context evolves.
  • Increased cost: Conducting RAI assessment for test cases can introduce additional development efforts and cost.

Related patterns:

  • RAl acceptance testing: The ethical quality of all the test cases designed for ethical acceptance testing should be assessed.

Known uses: