Continuous Documentation Using Templates

Summary: Continuous documentation is important to track development activities and ensure that the continuously evolving AI systems are responsible.

Type of pattern: Governance pattern

Type of objective: Trustworthiness

Target users: Project managers

Impacted stakeholders: Development teams

Lifecycle stages: All stages

Relevant AI ethics principles: Human, societal and environmental wellbeing, human-centered values, fairness, privacy protection and security, reliability and safety, transparency and explainability, contestability, accountability

Mapping to AI regulations/standards: EU AI Act, ISO/IEC 42001:2023 Standard.

Context: AI system development, particularly AI model development, is often performed in a fast-paced and experimental manner. Developers may prioritize writing code, rather than keeping detailed documentation of their work. For example, when developing AI models, developers usually use computational notebooks to combine code and notes together in a single document. It is challenging to communicate the ethical characteristics of AI systems.

Problem: How can stakeholders keep track of the development and use of AI systems and facilitate communication?

Solution: Continuous documentation is important for tracking development activities and ensuring that continuously evolving AI systems are responsible. The behavior of AI systems is determined by autonomous and potentially opaque AI models that are trained on data. Therefore, project teams need to create and continuously update documentation for the key artifacts of AI systems that may lead to ethical issues, such as data and models. There have been various types of templates for documenting data and models, such as model cards, data statements, datasheets for datasets, AI service factsheets, and data statements. Additionally, traditional software documentation needs to be extended to consider ethical aspects. For example, data requirements and ethical requirements should be explicitly specified in the requirement specification, while the results of ethical acceptance tests should be included in the testing report.

Benefits:

  • Provenance: Continuous documentation is essential for keeping track of the complete history and evolution of AI systems’ artifacts, such as data and model, over time.
  • Trust: Documentation is important in clarifying the context in which AI systems are intended to be used—for example, the situations in which they can be trusted to work well.

Drawbacks:

  • Time consuming: Continuously preparing documentation can be time-consuming and may divert resources away from other important tasks, such as coding and testing.
  • Error-prone: Documentation is often created and maintained manually, which can be error-prone. For example, the documentation may not be fully consistent with the code.

Related patterns:

  • Customized agile process: Documentation is important for recording the activities of a customized agile process, e.g., user acceptance testing.

Known uses:

  • Google provides model cards to enable transparent model reporting on model usage, provenance, and ethical evaluation.
  • Microsoft designs datasheets for datasets that allows every dataset to be accompanied with a datasheet documenting its motivation, creation, composition, intended uses, and other information to improve transparency and accountability.
  • IBM develops AI service factsheets to maintain AI services’ purpose, performance, safety, security, and provenance information.