Where AI Bias Actually Comes From
It seeps in at all three stages — pre-training, fine-tuning, and RLHF
① "AI bias" isn't one thing. People often lump together several distinct problems — representation bias, stereotype bias, and language/cultural bias among them.
② Bias isn't a bug — it's a direct result of the training process itself. It enters differently at all three stages: pre-training, fine-tuning, and RLHF.
③ "Just train on more diverse data" is only half right. In practice, there's no established best-practice fix yet — more data and correction attempts both tend to introduce new problems of their own.
① Bias isn't one thing — start by splitting it apart
"The AI is biased" actually points to several different phenomena. Lump them together and both the causes and the fixes get tangled.
| Type | What it is | Example |
|---|---|---|
| Representation bias | Certain groups, languages, or regions appear less often in training data | Overwhelming English text share means weaker performance in other languages |
| Stereotype bias | Occupations, gender, or ethnicity get statistically tied to specific traits | Defaulting "doctor" to male and "nurse" to female |
| Language/cultural bias | One culture's norms and assumptions get treated as the "default" | Date formats, family structures, and holidays described by Western norms |
| Political/viewpoint bias | Answers on contested issues lean toward one side | A two-sided debate gets described favoring only one position |
This article traces where and why each of these four arises, following the three-stage pipeline (pre-training → fine-tuning → RLHF) covered in how an AI model gets built.
② Pre-training — the internet itself is skewed
Pre-training does exactly one thing: solve "guess the next token" trillions of times over internet-scale text. Grammar, knowledge, and reasoning fall out of that process as byproducts — and bias rides along as a byproduct too, in exactly the same way.
The problem is that internet text isn't a balanced sample representing all of humanity. Text from one language (English), one era, and groups with better internet access dominates by a wide margin. And that text carries whatever statistical skews actually existed historically — records from eras when certain occupations really were dominated by one gender, for instance.
The model's only goal is to accurately predict the statistical patterns in its training data. If the data contains a pattern where a certain pronoun appears more often near a certain occupation, the model learning that pattern precisely is a result of accuracy, not intent. It's the same structure as hallucination arising without any "intent to lie."
A real example illustrates this. Between 2014 and 2017, Amazon built an AI tool to screen job candidates by training it on a decade of résumés — and that decade of résumés skewed heavily male. The model learned that pattern faithfully, downgrading résumés containing the word "women's" (as in "women's chess club captain") and penalizing graduates of women's colleges. Amazon tried patching specific terms, but the model kept finding new ways to reproduce the same discrimination, and the tool was eventually scrapped entirely (see the AI Incident Database record).
③ Fine-tuning and RLHF — bias enters at the moment a human picks
Fine-tuning refines the model on question-answer pairs people write and review; RLHF refines it by having people rank which of several answers is better. Both stages differ from pre-training in that a person is directly involved — but that person isn't perfectly neutral either.
If the people writing fine-tuning examples, or the raters comparing RLHF outputs, skew toward a particular language, age group, or profession, the style that group finds "good" hardens into the model's default personality. Flip the framing in "RLHF decides the model's tone and manner" around, and whose standard set that tone becomes exactly the channel through which bias enters.
④ Why "just add more data" doesn't solve it
The obvious response is "train on more diverse data." That genuinely helps, but it's not a complete fix, for two reasons.
Even if you could assemble text that perfectly represented all of humanity, if the reality that text describes was itself unequal, that inequality survives as a statistical pattern. Data representativeness and real-world imbalance are separate layers — fixing one leaves the other exactly where it was.
Second, the correction itself can create new failures. In February 2024, Google tuned Gemini's image generator to force racial diversity into its outputs — and it ended up randomly reassigning the race of historical figures who actually belonged to a specific ethnicity, like "1943 German soldiers" or "America's founding fathers." Google paused the feature and admitted on its own blog that "without some guidance, tools will naturally generate stereotypical images based on the data they are trained on" (Google's official blog). It's a case study in how a correction meant to reduce bias can simply shift the problem into a new kind of inaccuracy.
How much bias exists, and what the best fix looks like, are still active areas of research and debate. Be skeptical of any claim that a given model "has no bias." The opposite stance — "AI is just inherently biased, so it's not worth worrying about" — is an equally lazy conclusion.
⑤ How it actually shows up — patterns you're likely to notice
A few forms are especially noticeable for non-English-speaking users.
| Symptom | Cause |
|---|---|
| Non-English answers are less accurate or more awkward than English ones | English overwhelmingly dominates training data — the same root cause as higher token costs for other languages |
| Date/address/name formats and holiday or family descriptions default to Western norms | Western text makes up a disproportionate share of training data |
| Occupations or roles default to a particular gender | Stereotype bias — a statistical pattern from the pre-training stage |
Regulators are moving to keep this from going unaddressed. Since 2023, New York City has required companies using AI hiring tools to undergo an independent annual bias audit, and by late 2025 real fines had already been issued to companies that skipped one (NYC's official guidance). The EU AI Act likewise classifies AI used in hiring and credit scoring as "high-risk," imposing separate obligations (EU AI Act guidance). That's a signal that AI bias is being treated as a practical, legal risk — not just a theoretical concern.
⑥ A practical checklist
□ For high-stakes decisions — hiring, lending, evaluations — treat an AI answer as a reference, not the final call, and have a human make the final review.
□ On contested topics, deliberately ask for the opposing view to check whether the answer leans one way.
□ Notice when a "default" assumption (gender, culture, format) is genuinely neutral versus simply the statistically common one.
□ For sensitive judgment calls, cross-check with a different model or a differently worded prompt.
The short version: AI bias is less a "bug to fix" and more a property that needs ongoing management as long as the model learns from human-made data. Knowing where it comes from at least tells you which answers deserve more scrutiny.
※ This article is based on publicly documented cases and research as of September 2026, and does not criticize any specific company or product or constitute investment advice. Regulatory status can change — verify current requirements against each agency's official materials.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.