Chapter 4. Bias, Discrimination, and Unequal Outcomes

4.1 How bias enters an AI system

AI can reproduce or amplify patterns in historical data, labels, institutional practices, and design decisions. If past hiring records reflect unequal access to opportunity, a model trained to imitate those decisions may reproduce the pattern. If a dataset under-represents a language or community, the system may perform worse for that group. Bias can also enter through the choice of target variable: a measurable proxy such as prior spending, recorded complaints, or historical performance may not measure the underlying concept fairly.

Fairness is not a single technical score. Different fairness criteria can conflict, and the appropriate assessment depends on the use case, legal context, and affected people. A model can have similar average accuracy across groups while still creating unacceptable errors for a subgroup or imposing a disproportionate burden on people with less power.

4.2 The difference between equal treatment and fair outcomes

Treating every person identically can still be unfair if the system ignores relevant differences in access, disability, language, or circumstances. Conversely, tailoring a service can be legitimate when it is evidence-based, lawful, proportionate, and designed to improve access rather than stereotype people. Fairness analysis must begin with the purpose of the system and the real consequences for affected individuals.

AI should not be used to disguise a human decision as neutral mathematics. A score can appear objective while encoding contestable assumptions. Organisations should explain the intended purpose, data sources, limits, error rates, and available remedies.

4.3 Practical fairness controls

Define the legitimate purpose and prohibited uses before selecting a model.
Check data coverage, label quality, missingness, and historical distortions.
Test performance across relevant groups and real operating conditions, using lawful and privacy-protective methods.
Include affected users and subject-matter experts in design and review.
Provide accessible explanations, appeal channels, and human reconsideration.
Monitor for drift and unequal outcomes after launch; do not assume a one-time test is sufficient.
Keep a record of decisions, trade-offs, limitations, and remediation.

4.4 Socio-economic perspective

Unequal access to devices, reliable connectivity, language support, digital skills, and paid AI tools can widen existing divides. Those with the resources to verify, customise, and supervise AI may gain more than those who must accept automated outcomes. Public policy and institutional practice should therefore include accessibility, affordable access, training, local-language capability, and protections for people who cannot reasonably opt out.