Ask an AI chatbot a question and it will answer. Clear. Organized. Sure of itself.
Most of the time, it’s right. Some of the time, it is making the whole thing up — and it sounds exactly as sure either way.
That last part is the problem. Not that the machine makes mistakes. That it mistakes them.
So basically, it’s autocomplete with a really good memory
Here’s the one idea that makes everything else click.
Your phone keyboard guesses the next word when you type. “See you” becomes “See you tomorrow.” Your phone doesn’t know your plans. It just knows that “see you tomorrow” is a common pattern.
A chatbot is the same trick at a scale that’s hard to picture — trained on more text than you’ll read in a hundred lifetimes, and way smarter about context. Most of the time, the most likely next words are also the true ones. That’s why it feels like it knows things.
But likely and true are not the same thing. And that gap has a name: hallucination.
Here’s where it gets interesting: sounding right and being right are different skills, and the machine was only ever trained on the first one. OpenAI’s researchers have noted that big models almost never misspell words. Grammar, tone, the shape of an answer — those are strong patterns. The model learned those perfectly. Whether the content is true? That was never its job.
Think of it like a student who didn’t study for the exam but knows exactly what exam answers are supposed to look like. The handwriting is neat. The structure is perfect. The paragraphs flow. The facts are invented.
— Roxie: This isn’t a bug that got fixed in the latest update. It’s how the machine is built. Every model you use, from every company, does some version of this. The ones that pretend otherwise are the ones you should trust least.
The confident lawyer and the $5,000 fine
You know this is real, not theory. It keeps happening to people who should have known better — which is exactly why the rest of us need to take it seriously.
Back in 2023, a New York lawyer used ChatGPT to help write a legal brief. The tool produced six court cases that backed his argument perfectly. Names, citations, quotes, everything formatted the way judges expect. None of them existed. When he started to doubt them, he asked the chatbot whether they were real. It told him yes. A judge fined him, his co-counsel, and their firm $5,000 — and the story became the cautionary tale every law school now teaches.
It is not just lawyers. In the summer of 2025, a reading list ran in real newspapers — including the Chicago Sun-Times — and several of the books on it didn’t exist. The titles had been invented by AI and attached to real, well-known authors. The writer had used a chatbot for research and hadn’t checked the results before they went to print. It went out in a newspaper. On paper.
And the pattern keeps showing up in places where it costs real money: a tribunal held Air Canada responsible for a refund policy its chatbot invented — and rejected the airline’s argument that the bot was responsible for its own words. New York City’s official business chatbot told shop owners they could take workers’ tips and refuse Section 8 vouchers — advice that would have broken the law. A California dealership’s chatbot got tricked into agreeing to sell a new Tahoe for one dollar.
— Roxie: Notice the common thread. In every story, the output was fluent, specific, and formatted exactly right. The formatting is the trap. Your brain sees a citation and relaxes. That’s the part to unlearn.
Why fluency fools you (and why that’s actually normal)
Your brain is doing something sensible, by the way. Humans have spent our whole lives learning that confident, fluent, well-structured speech usually comes from someone who knows what they’re talking about. That’s been true for thousands of years.
It’s just not true of a machine that generates text one word at a time with no fact-checker running in the background.
OpenAI’s own explanation uses an analogy I really like. Train a system on millions of pet photos labeled “cat” or “dog” and it learns to tell them apart just fine. Now label those same photos with each pet’s birthday. It will always get it wrong, because nothing in a photo tells you someone’s birthday. That’s what they call an “arbitrary low-frequency fact” — a detail that rarely shows up in the training text and can’t be figured out from anything else. Asked for one co-author’s dissertation title, a widely used chatbot confidently gave three different answers. None were correct.
When the pattern isn’t there, the machine doesn’t stop. It can’t — stopping isn’t in its vocabulary unless someone teaches it. It just gives you the most likely-looking answer and delivers it with full confidence.
The five reasons it gets things wrong
Hallucinations aren’t random. They show up in the same situations, over and over. Once you see the pattern, you can spot a risky answer before it costs you.
- It never saw the answer in the first place. Some topics just barely appear in training data. The model fills the gap with patterns from similar topics. General ideas hold up fine; obscure specifics don’t.
- Its knowledge is out of date. Every model has a cutoff — training ends, the world keeps going. Ask about something newer than the cutoff and you’re asking it to guess. It will, and it won’t warn you.
- It was trained to be helpful, not to be honest. Saying “I don’t know” was never the behavior that got rewarded. So when it’s unsure, it gives you something. Helpful-looking. Confident. Wrong.
- Exact details are where the fabrications hide. Numbers, dates, quotes, names, links. The more specific the fact, the more likely it’s invented. General explanations are usually safe. Precise figures are not.
- You led it there. Leading questions get led answers. Ask “why did Einstein win two Nobel Prizes?” and some models will happily explain — even though he won one. The model follows your framing instead of correcting it.
— Roxie: Number 3 is the one that matters most. These models are optimized to sound like they helped you. Sometimes the most helpful thing is a refusal, and most models had to be taught to do it. The ones that learned to say “I’m not sure” — MIT researchers showed this works with reinforcement learning back in April 2026 — are worth your trust. A model that admits its limits is infinitely more useful than one that confidently feeds you fiction.
What actually moves the needle
So what do you do about it? Not panic. Not quit using the tools. Just change the relationship a little — from “the AI said it” to “the AI drafted it, and I checked it.”
The single most effective fix is embarrassingly simple: stop asking the model to answer from memory alone. Give it the source first. If you’re asking about your company’s policy, paste the policy. If you want it to summarize a paper, upload the paper. When the model works from a document instead of its memory, fabrication rates collapse — one analysis found reductions of 30 to 70% across domains, and OpenAI’s own evaluations show rates dropping under 2% on source-grounded tasks. It can’t invent what it doesn’t need to invent.
A few more moves that actually work, ranked by effort:
- Tell it to show its work and quote its sources. “Quote the passage you’re basing this on” forces the model to point at something real — or admit there’s nothing to point at.
- Add three words to your system prompt or custom instructions: “admit when unsure.” Explicit instructions to say “I don’t know” rather than guessing measurably reduce fabrication. It’s the cheapest fix there is.
- Verify the high-stakes stuff independently. Anything involving money, medicine, law, or anything you’ll sign your name to — check it against the primary source. A thirty-second search costs less than a $5,000 fine.
- Watch the specific details, not the vibe. When you’re reading an AI answer, skim past the confident tone and interrogate the facts: the numbers, the names, the quotes, the links. That’s where the fabrications live. If a link doesn’t open, that’s your tell.
- Keep it away from the deep end. The safest use of AI on consequential decisions is as a first draft that a qualified human verifies — not skims, verifies. That defeats some of the automation fantasy, but the alternative is a chatbot making policy for your business.
— Roxie: No method eliminates hallucinations. Zero. Anyone selling you a model or a tool that “never hallucinates” is hallucinating about hallucinations. The user is the last line of defense, which means the informed user wins. That’s you, now.
The verdict: trust the draft, verify the facts
Here’s the thing. None of this means the tools are broken or useless. I use these things all day, every day. The free-first guides on this site exist because the tools are genuinely good at a lot of work.
It means the machine is a word-predictor, not a truth-teller — and once you know that, it’s easy to work with. Give it sources. Ask it to show its work. Check the specific facts. And when the stakes are real — your money, your legal filings, your company’s name on the answer — give it thirty seconds of verification before you act on it.
The fake reading list, the fake court cases, the one-dollar Tahoe — none of those happened because the AI was bad. They happened because someone treated a confident draft like a verified answer.
So basically: let the AI be confident. You be the careful one.
Keep learning: How to Prompt AI Tools So They Actually Give You What You Want · The 5 Prompt Patterns That Fix 90% of Bad AI Output · AI Agents Explained: What They Actually Do (And the 5 Worth Trying)