Wondering whether AI really treats people unfairly? These AI bias examples show it can: documented cases in healthcare, hiring, criminal justice and face analysis, where systems trained on historical data reproduced old inequalities.
If you are new to the field, start with the guide to what is artificial intelligence, then come back for the cases below.
| Item | Value |
|---|---|
| Case studies in the reviewed preprint | 4 case studies |
| Amazon recruiting tool built and tested | 2018 |
| HUD final disparate impact rule released | September 2020 |
| Coalition that opposed the HUD proposal | 23 organizations and experts |
| Routes of bias named by the preprint | 3 (training data, proxy variables, optimization objectives) |
These figures come from a Research Square preprint and a New America report. They can change, so check the source before you rely on them.
Why do AI systems pick up bias?
A review posted on Research Square, which is a preprint and has not been peer reviewed by a journal, traces bias to the way machine learning pipelines are built. It names these routes:
- skewed training data
- poorly chosen proxy variables
- the design of optimization objectives
New America makes a similar point about training data: predictive algorithms are designed from correlations found in data of past experiences. If historical data show that in the past only white men held certain jobs, those patterns can be perpetuated by the algorithm.
Bias can also appear when a dataset has no sensitive categories at all. Highly correlated variables can serve as proxies for race or gender, so deleting a column does not remove the problem.
If terms like models and training data are unfamiliar, this explainer on machine learning vs deep learning covers the background.
Healthcare: a risk-scoring tool that missed Black patients
The preprint’s four case studies begin with Obermeyer et al.’s analysis of a commercial healthcare risk-scoring tool. New America summarises the finding this way: the system failed to identify Black patients at risk for medical needs at the same rate as white patients.
The result was that Black patients were less likely to receive preventative care to improve their health. The preprint describes the same pattern as an algorithm that routes fewer resources to Black patients despite equivalent or greater medical need.
Hiring: the Amazon recruiting tool
In 2018, according to New America, Amazon built and tested an AI recruiting tool to help find and review job applicants. Amazon found that the tool preferred male candidates over female candidates for technical positions.
The tool was not built to consider gender. It was built to identify candidates from patterns in previous resumes, and applications for technical positions had historically come predominantly from men. Amazon abandoned the tool before it was used to recruit candidates.
New America draws a practical lesson from this case: AI systems need testing and auditing for discriminatory impacts before and after deployment.
Criminal justice, face analysis and housing
The preprint also covers ProPublica’s investigation of the COMPAS recidivism tool. It describes a tool that assigns a Black defendant a higher risk score than a white defendant who later reoffends, and says this is not random noise.
The fourth case is Buolamwini and Gebru’s audit of commercial facial analysis systems. The preprint describes a system that misidentifies a dark-skinned woman while correctly classifying a light-skinned man.
Housing is a policy example rather than a single tool. New America argues that the final rule the Department of Housing and Urban Development (HUD) released in September 2020 puts a heavy evidentiary burden on plaintiffs, even though those affected often lack knowledge of how the tools work. Its author organisation filed comments as part of a coalition of 23 civil rights and consumer advocacy organizations and individual experts.
What reduces bias, and what does not
The preprint reports that competing definitions of fairness are mathematically incompatible, so no single algorithm can satisfy all of them at once. Choosing a definition is therefore a trade-off, not a purely technical step.
Its conclusion is that technical fixes alone are not sufficient. Durable fairness, it says, requires:
- institutional accountability
- representative data governance
- enforceable legal standards
The review also looks at regulatory frameworks, including the EU AI Act and New York City’s Local Law 144. New America adds that decision-makers often trust automated systems because they seem less biased than people, which is why testing matters.
If you want to examine models yourself, this path on how to learn machine learning is a sensible place to begin.
Frequently asked questions
What is algorithmic bias?
Show answer
It is a pattern where an automated system produces systematically unfair outcomes for some groups. The preprint says AI systems trained on historical data reproduce historical inequalities, and in some cases widen them.
Does removing gender or race from the data fix bias?
Show answer
Not necessarily. New America notes that discrimination can occur when a dataset contains only highly correlated variables that serve as proxies for those characteristics. Amazon’s tool was not built to consider gender, yet it still preferred male candidates.
Why do people trust biased systems?
Show answer
New America says there is often a perception that automated systems are less biased than human decision-makers because they rely solely on data and statistics. Testing often finds unintended, biased results.
Can one algorithm satisfy every fairness definition?
Show answer
According to the preprint, no. It reports that the formal definitions are mathematically incompatible, so no single algorithm can satisfy all of them simultaneously.
