Fairness Assessor

Summary: Selecting appropriate metrics for measuring the fairness of AI models is crucial for detecting and mitigating fairness risk in AI systems.

Type of pattern: Product pattern

Type of objective: Trustworthiness

Target users: Data scientists

Impacted stakeholders: RAI governors, AI users, AI consumers

Lifecycle stages: Design

Relevant AI ethics principles: Fairness

Context: In the context of AI systems, unfairness refers to the situations where the outcomes of decisions and behaviors of the systems lead to discrimination against certain groups of people, such as those based on genders, races, and other protected characteristics.

Problem: How can we quantitatively measure the degree of bias for different types of fairness in AI systems?

Solution: When detecting and mitigating bias in AI models, it is critical to select the appropriate fairness metrices for measuring fairness. Some commonly used fairness metrics include: 

  • Demographic parity: This metric measures the likelihood of a positive outcome (such as getting a job) across different sensitive groups (such as gender) and ensures it is the same for all groups. For example, an AI model meets the conditions of demographic parity if 10 percent of males are predicted as positive, and the same ratio applies to females. Demographic parity is used to prevent historical biases and can be achieved by modifying the training data.
  • Equalized odds: This metric measures the equality of true positive and false positive rates between different sensitive groups (such as gender or race). For example, an AI model satisfies the conditions of equalized odds if the rate of a qualified applicant being hired and the rate of a disqualified applicant not being hired are the same for both males and females. Equalized odds should be used when a false positive outcome is costly and there is a strong need for true positive outcomes. Also, there should be a clear decision threshold for the target variable (e.g., determining fraudulent transactions).
  • Equal opportunity: Each group of sensitive attributes should have equal true positive rates. For example, an AI model satisfies the conditions of equal opportunity if qualified loan applicants have an equal chance of getting a loan regardless of the suburb they live in. Both equalized odds and equal opportunity can be achieved through post-processing mechanisms. Equal opportunity is typically used when there is a strong need to correctly predict positive outcomes, and when introducing false positive outcomes does not have a costly impact on the users of the AI systems. Additionally, the target variable should have a clear decision threshold that is not subject to bias.
  • Fairness through awareness: An AI model should produce similar predictions for individuals who are similar with respect to a task-specific similarity metric. This metric provides fairness guarantees at the individual level rather than at the group level. To achieve this, a distance function must be defined to measure similarity between individuals.


  • Quantifiable measurements and validation: The use of fairness metrics allows for the quantification of bias, making it easier to identify and validate over time.
  • Monitorability: Fairness metrics can be continuously monitored at runtime to detect deviation from fairness requirements.
  • Compliance: By measuring the bias, the fairness assessor pattern can help organizations comply with laws and regulations related to fairness and discrimination.


  • Limited by access to demographics: In real-world scenarios, demographics information is often available only at a coarse level, making it challenging to accurately measure bias at a more granular level.
  • Restricted by application type: While quantitative fairness metrics are effective for certain types of applications such as automatic recruitment, face recognition, and crime prediction, they may not be suitable for applications involving frequent human-AI interactions.
  • Reductionism: Fairness is a complex and evolving social concept that is heavily dependent on context and can be quite subjective. Reducing fairness to metrics can oversimplify the multi-faceted nature of the concept and may mask important issues that are not captured by the metrics.
  • Conflicting metrics: Many fairness measures are inherently in conflict with each other, making it difficult to satisfy all desirable measures of fairness simultaneously. For example, achieving demographic parity might lead to a lower true positive rate for certain groups, while achieving equalized odds might result in a lower true positive rate for other groups.

Related Patterns:

  • Discrimination Mitigator: Fairness measurement metrics can be used to measure the effectiveness of discrimination mitigation techniques.

Known Uses:

  • Google Fairness Gym is a simulation toolkit that analyses the fairness of AI models with consideration for the context (e.g. temporal and environmental context).
  • Google Fairness Indicators is a library that enables computation of popular fairness metrics.
  • Facebook Fairness Flow is a fairness toolkit used by Facebook internally to identify bias in AI models.