IntegrAuth Academy
Identity, security and practical AI — free, from zero. Byte-sized lessons, real diagrams, hands-on labs, zero fluff. Two programs, each with its own exam and certificate.
English is hard? Your browser can translate this page into Tamil, Telugu, Kannada, Malayalam or Hindi. Tech words stay in English so you learn the real names.
How?
- Chrome on phone: ⋮ menu → Translate
- Chrome on computer: right-click → Translate to …
- Edge: the translate icon in the address bar
Bias and fairness
Priya gets 2,000 job applications. An AI tool offers to pick the best 50. Your goal: test if an AI treats people fairly.
🧪 Interactive lab — enable JavaScript to play with this one.
- An AI learns habits from past data. If you let it decide, its habits become your decisions.
- Test what comes out, not only what goes in.
- For big decisions about people, the AI should help. A person should decide.
Where bias comes from
An AI finds patterns in past data. It can't tell fair patterns from unfair ones. So it can learn to favor people because of who they are. This is algorithmic bias.
Deleting a field like sex does not fix it. Other details can point to it, like the word "women's". A graduation year can point to age. Such a detail is a proxy variable.
The swap test
You don't need to see inside the AI. Change only one detail that should not matter, like a name or a postal code. Keep the rest the same. If the score moves, the AI uses that detail. This is a counterfactual test.
Answers vary from run to run. So run each pair several times. Trust a gap only if it is bigger than the normal spread.
Count the outcomes
Also count how often each group gets picked. Divide the lower rate by the highest rate. This is the impact ratio. Say 30% of women pass and 50% of men pass. 30 ÷ 50 = 0.6. US hiring guidelines have the four-fifths rule. A ratio below 0.8 is generally seen as a sign of unfair impact. Compare rates, not head counts. The same count works for a loan app: how many village applicants get a loan, compared with city applicants?
"Fair" can mean more than one thing. Build a shortlist and see.
🧪 Interactive lab — enable JavaScript to play with this one.
Go deeper: where bias enters
- The data. In 2018, Reuters reported that Amazon dropped a test tool that rated job applicants. It learned from ten years of résumés, mostly from men. It marked down résumés with the word "women's". Amazon said recruiters never used it to judge candidates.
- The labels. A 2019 study looked at a tool US health systems used to pick patients for extra care. It used health costs as a stand-in for health needs. Less had been spent on Black patients with the same needs. So Black patients were sicker than white patients with the same score.
- The use. The 2018 "Gender Shades" study tested three tools that guessed gender from face photos. Errors reached 34.7% for darker-skinned women. For lighter-skinned men, they were never above 0.8%.
Go deeper: two harms
An allocational harm happens when a system hands out chances or resources unfairly. Think of jobs, loans, housing or medical care. The person rarely learns that a machine marked them down.
A representational harm happens when a system shows a group unfairly. It may use a stereotype. It may fail to recognize the group at all. In a 2023 Bloomberg study of AI images, women rarely appeared as doctors, lawyers or judges.
Go deeper: meanings of fair
| Check | The question it asks |
|---|---|
| Equal selection rates | Is each group picked at the same rate? |
| Equal error rates | Do qualified people in every group have the same chance of being picked? |
| Calibration | Does a score of 7 mean the same thing for every group? |
You can't always meet all three at once. Say real rates differ between two groups. Then a score usually can't be calibrated and have equal error rates. So choose what "fair" means for each decision. Write it down before launch.
Go deeper: swap tests
Priya's team tested forty résumés. Swapping names moved scores no more than normal run-to-run changes. But a career gap cost ten points. More of their applicants with gaps were women. A vendor's test on other people's data does not prove a tool is fair on yours.
The four-fifths rule is a first check, not a final answer. On why runs differ, see Why answers change: temperature.
Go deeper: keep humans in charge
In hiring, lending, housing, health care and justice, AI should help with a decision, not make it. The law often agrees.
- In the US, laws against unfair treatment still apply when software does the sorting. In 2023, iTutorGroup agreed to pay $365,000 to settle a case. Its software was said to reject older applicants automatically.
- The EU AI Act treats AI for hiring and credit scoring as high-risk. It requires human oversight. See AI rules at work.
- In the EU, the GDPR limits big decisions made only by a machine. Where such decisions are allowed, people can ask for a human.
A human review only helps if it is real. The reviewer needs time, the power to overrule and the reasons behind each score. Otherwise people just agree with the machine. See Staying sharp with AI.
Ask an AI to rate a made-up résumé from 1 to 10. Do it 5 times, in new chats. Then change only the name. Compare the averages.