Classroom Edition #4: Why Do AI Models Hallucinate?

You may be able to test this in about a minute.

Upload a five-page document to a weaker model, or an older ChatGPT model, and ask it a question about page seven. Sometimes it will catch you. Sometimes it won't, and you will get specific details about a page that doesn't exist.

These are called hallucinations, and they are definitely frustrating... but also misunderstood.

I've mentioned hallucinations in multiple newsletters, but they're important to understand, and they are a core property of how LLMs work. So I am going to do a deeper dive this week.

The example above is simple, and hallucinations have changed over the last couple of years. It would likely be difficult to get ChatGPT to make this mistake today. Today's hallucinations are typically more subtle.

A hallucination is when a model tells you something false in a confident tone. It's the primary reason you can't trust models.

It is not technically lying to you. Lying requires knowing the truth and deliberately saying something else.

When a model hallucinates, it isn't trying to deceive you. It simply doesn't have enough information to answer correctly, so it generates a response that sounds plausible.

It is also not an "error mode" the model falls into. It is actually caused by the same thing that makes these tools useful. In 2021, Nature published a paper called "De novo protein design by deep network hallucination." Researchers had a network invent protein structures that had never existed, then built 129 of them in bacteria. Twenty-seven of them folded exactly the way the network said they would. One fifth of what it invented turned out to be real, working protein.

The ability to produce something plausible that has never existed is the product. That's the main reason people use LLMs. You cannot remove what is core to a model. You can only take measures to ensure you are aware when it is happening.

Three reasons, and they build on each other.

One. It predicts, it does not look up.

A large language model is not a database. There is no stored answer it looks for and retrieves. It reads the information that came before, predicts the next token (roughly a word, or part of one), then the next, one at a time, and it does this very fast... typically.

What it produces is the most likely continuation of the text. Usually the most likely continuation is also true, which is why it does cool things.

Here's how that creates invented facts. Say I ask a model to write a program that needs a software package. If it can't remember the correct one, it may simply make up a package name that sounds real. It isn't checking whether the package actually exists. It's predicting what is most likely to come next.

This is not hypothetical. Researchers had sixteen models generate 576,000 programs. Those programs recommended 2.23 million packages, and 19.7 percent of the recommendations were for packages that never existed.

If you want to watch this happen, Transformer Explainer from Georgia Tech runs a small model live in your browser and shows you its candidate next words and how likely it thinks each one is.

Two. Some facts appear only once.

Training data is enormous but uneven. A common fact appears a million times and gets learned solidly. An obscure fact may appear once, or never, and there is nothing solid for the model to draw on.

The leading research on this topic found that this is an unavoidable consequence of pretraining. If a model doesn't have enough reliable information, it still has to produce an answer. That's why the authors argue the solution isn't just a better base model. It's what happens after pretraining.

Three. We grade models in a way that rewards them for guessing.

In April 2026, Nature published that paper by researchers at OpenAI and Georgia Tech, and its second argument points at us rather than the model.

Think about a multiple-choice test where a blank scores zero and a wrong answer also scores zero. The best strategy is to answer everything, including the questions you have no idea about. A student who leaves blanks loses to a student who guesses. Nobody teaches that student to bluff. The scoring does.

The authors then checked the benchmarks the industry actually uses to rank models, the ones quoted in every launch announcement. Of the ten main ones, nine give zero credit for answering "I don't know."

So the industry built scoreboards that punish uncertainty and trained models against those scoreboards.

There is one encouraging finding. Anthropic recently found evidence that its model starts from a cautious position when asked about a person it may not know. Only after it decides the name seems familiar does it begin generating an answer. The researchers believe some hallucinations happen when that recognition step goes wrong. The model thinks it knows the person, even when it doesn't, and confidently fills in the gaps. Anthropic describes this as a hypothesis, not a proven explanation.

If you learned about this in 2023, you learned to watch for invented sources. A court case that does not exist, a book nobody wrote.

That is no longer the main failure. Deep research modes, the ones that go away for several minutes and come back with a report and a list of sources, have mostly stopped inventing sources outright. An audit this year found that for the top models, over 94 percent of the links work and over 80 percent of the pages are on the right topic. But the share of citations that actually support the specific sentence they are attached to runs from 39 to 77 percent.

You were told to check that the source is real. The source is real. The link opens. The page is about the right subject. It just does not say what the sentence claims it says.

That is what makes this version more dangerous. The old failure was easily catchable by anyone willing to click on a link. This one is only catchable by someone willing to read and manually verify.

In August 2025, Deloitte delivered a A$439,000 report to an Australian government department. It quoted a Federal Court judge, Justice Jennifer Davies, from two paragraphs of a judgment. Her name was spelled wrong. And the quotes were never hers. The report also cited academic papers nobody had written, and invented a second judge who didn't exist.

In May 2026, EY pulled a report after 16 of its 27 citations turned out to be fabricated, misattributed or broken. In June 2026, KPMG withdrew a report after an outside analysis found only 5 of its 45 citations checked out.

A researcher at HEC Paris maintains a public database of court filings caught containing fabricated citations, searchable by country and outcome. It holds 1,822 cases, up from 87 when it launched in May 2025.

One honest note about that database... going through its case records, more than half of those filings came from people representing themselves without a lawyer. This is probably less a story about professionals being replaced, and more about what happens when someone uses a chatbot as a substitute for expertise.

Air Canada is another famous example. Its support chatbot told a customer they could be reimbursed for travel and later claim a bereavement discount. That policy never existed. The customer relied on the chatbot's answer, the airline refused to honor it, and a tribunal ordered Air Canada to pay. When your AI invents a policy, your customers don't care that it was "just a hallucination."

Asking a model to recall obscure facts from memory, with no web search function turned on, is a terrible idea. Grounding your chatbots with documents or web sources fixes most of that. The term grounding just means the model has to answer from something in front of it rather than from memory. On the benchmarks where OpenAI ran the same model with search on and off, turning it on cut factual errors by two to four times.

But it does not go to zero, and this is the number worth remembering. On OpenAI's own measurement of real traffic, with search on, about 4.5 percent of the claims in an answer are still wrong. Roughly one claim in twenty. And it typically looks exactly like the other nineteen.

Risk also climbs the longer the machine works unsupervised. In that citation audit, how well sources supported their claims degraded by about 42 percent as the number of steps went from 2 to 150. A quick grounded answer is fairly safe. A twenty-minute autonomous research run is not, and it arrives looking far more authoritative and believable.

Turn search on for anything factual. This is the largest single improvement available to you. It is on by default in most consumer products now, which means the thing to actually check is whether it stayed on: long conversations, uploaded files and custom setups can quietly leave the model answering from memory.

Open the source and read the sentence. Not "does the link work." Not "is this the right paper." Find the sentence the claim came from. You will not do this for forty citations, so do it for the two or three that matter the most.

Ask for the case against the answer. This does not make the model more accurate. It makes you better at judging it. In a study this year, people shown one-sided reasoning were about 1.7 times more likely to accept a wrong answer than people shown both sides, and they rated the one-sided version as more helpful.

Stop using pointless "magic phrases." "Do not hallucinate." "Only use verified facts." I went looking for controlled evidence that these help and found none. The same goes for "are you sure? double-check that." In the study that tested self-correction with no new information, on an older model, 7.6 percent of answers went from wrong to right and 8.8 percent went from right to wrong. Accuracy fell.

And one thing to stop trusting. Citations make you feel better whether or not they are any good. A study presented at AAAI last year found that showing people sources raised their confidence in an answer even when the sources were irrelevant ones pulled from someone else's question. Checking them was what brought that confidence back down. The name for this is automation bias.

Same problem, better tools, because code can be run to automate some verification.

Make it run the code and show you the output. An agent that writes code and hands it to you has produced something that looks right. An agent that runs it and shows you what happened has produced evidence.

Do not accept "I tested it" as testing. Transluce found 71 cases where a pre-release version of OpenAI's o3 claimed it had run tests on a MacBook it never used, then doubled down when challenged. Ask for raw output, logs, or screenshots, not a summary. And do not assume a second model will catch it: in one benchmark, the best model found another agent's failed tool call only 11.6 percent of the time.

Be careful with package names it suggests. With one in five recommendations naming something that does not exist, attackers have started registering the fake names and putting real code in them. A researcher this year found 127 names that five different models all invented identically; 53 were still unclaimed and registrable. Checking that the package exists is not enough, because the dangerous ones now do exist. Check how old it is, how many people use it, and whether it has a real project behind it.

I was researching example for this article and it hallucinated four times, I only caught them because I went and read the sources it provided.

The draft mentioned researchers generated 2.23 million code samples and that 19.7 percent of them recommended a fake package. The actual study generated 576,000 samples, which contained 2.23 million package recommendations, and 19.7 percent of those were fake. The number in my first draft appears nowhere in the paper. I nearly published an invented statistic in an article about invented statistics.

The first draft also mentioned Transport for London, NHS Greater Manchester, UBS and Swiss Federal Railways all said the projects in KPMG's report never happened. They never said that. The projects are real. What those organizations disputed was how KPMG described them: "factually incorrect," "not accurate," "misleading." My draft invented a denial and attached four real institutions' names to it.

Last week I had part of a chart reviewed twice, line by line, each time by a model with fresh eyes. Both reviews came back clean. The chart was drawing itself in the wrong position on the screen. The code read as correct and was correct in every way except the one that mattered. A screenshot caught it immediately.

Reviewing a description of the work is not reviewing the work. That holds whether the description was written by a machine or by a person.

The most important thing to understand about hallucinations today is not that these tools are confidently wrong. That much is understood, and it is slowly getting better.

It is that most of what is being sold to you as the fix makes you more confident without making you more correct. Citations you do not open. Reasoning traces that make a wrong answer feel more trustworthy. Tools and methods are coming out to make people feel more confident. But that just makes the problem worse, not better.

The features that look like verification are still mostly confidence features. The validation is your job.

Deloitte's problem was not GPT-4o's fault. It was that an A$439,000 report went out the door and the first person to open one of its citations... was not a Deloitte employee.

Go deeper:

You now understand hallucinations better than most of the people confidently explaining them on this platform.

Thanks for reading 🙏

🔗 LinkedIn: linkedin.com/in/dustinwcole

🌐 Site: dustincoledata.com

— Dustin