Retrieving and Generating, or: Where Answers Come From

This entry explains the two ways a computer can answer a question, and why one of them occasionally invents things with a perfectly straight face.

Fetchers: Search Engines and Catalogs

A search engine and a library catalog both answer questions the same unglamorous way: by finding something somebody stored earlier. These systems retrieve information, which means they find a stored item and hand it back unchanged. No improvements, no opinions. Google keeps information about web pages in its index, which Google describes as a large database. When a person types a query, the search engine finds matching pages in the index and returns links to them. A library catalog works the same way. Search for a book the library owns, and you retrieve the record for that book. Search for a book the library does not own, and there is no record to find, so you get nothing. This is disappointing, but it is honest. A library catalog cannot generate a record for a book it does not hold, and it would not dream of trying.

A note on Google. For travelers: here, a Google search means the list of actual websites below the search box. It does not mean the AI Overview that Google sometimes places at the top without being asked.

Makers: How a Language Model Generates

As the previous entry explained, a language model is two files: a large file of numbers called parameters and a smaller program called a runtime. Researchers have shown that large language models store factual knowledge in their parameters, smeared across billions of numbers. A language model can generate text, which means it produces new text by predicting it one token at a time. A token can be a whole word, part of a word or a single character. First, the runtime uses the parameters to work out how likely each possible next token is. Next, it picks one and adds it to the text. Then it does the whole thing again, and again, until the answer is finished. Ask a chatbot how a battery works, and the model will generate the explanation token by token, like someone laying track just in front of a moving train. A chatbot answering from its parameters does not retrieve anything, because there is no stored copy of the answer anywhere for it to find. The writer Ted Chiang points out that a chatbot cannot reproduce an exact quote from a web page, because the model keeps only an approximation of the text.

Hallucinations, Delivered Calmly

A model will generate text for almost any prompt, including prompts its parameters know nothing reliable about. It does not get nervous. A hallucination is generated text that sounds correct but contains a false detail. In a 2025 paper, researchers asked a model for the birthday of one of the authors and asked for an answer only if the date was known. Across three attempts, the model generated three different dates. All three were wrong, which is a lot of birthdays for one person. The model is not broken when this happens. It is predicting the next token exactly as usual, and this time the prediction happens to be wrong. The same paper explains that a birthday follows no pattern. When a fact like that appeared rarely in the training text, a model often cannot predict it correctly. A typo in a stored record is not a hallucination, because a person made that mistake while typing. The system will retrieve the record exactly as it was stored, typo and all, with the loyalty of a dog bringing back a shoe.

Chiang compares a language model to a blurry JPEG, a photo saved with lossy compression. In his comparison, a hallucination is like a compression artifact: a detail filled in where the original was not kept. Lossy compression is a useful way to think about a model, although it is a comparison, and comparisons always leave something out.

Tools That Do Both

Some tools fetch and then make. They search first, and then they generate an answer from what they found. The AI summary at the top of a Google search page works this way. Google states that its AI Overviews come from a language model connected to its ranking systems, which find relevant pages in its index. The search step retrieves pages, and the language model step generates the summary. The summary can still be wrong, and Google itself reported that some AI Overviews misread a question or treated a satirical web page as fact. Figure 1 shows both paths side by side, with a dotted path for a tool that does both.

The most common mistake in this part of the galaxy is copying an AI answer without checking it. The fix is one question, asked every time before you use an answer: "What source can I check this answer against?"

Summary

A search engine or a library catalog retrieves stored items and returns them unchanged, or it returns nothing. A language model generates new text from its parameters one token at a time, and a hallucination is generated text with a false detail, delivered calmly. Some tools search first and then generate, and any generated answer needs a source to check it against.

Sources: Google, In-depth guide to how Google Search works (2025); Google, Introduction to Large Language Models (2026); Library of Congress, Library of Congress Catalog (n.d.); Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020); Kalai et al., Why Language Models Hallucinate (2025); Chiang, ChatGPT Is a Blurry JPEG of the Web (2023); Reid, AI Overviews: About last week (2024).