Review 1: What a Model Is
- Terminology and concepts to look out for: training · lossy · prediction
- Relevance to me: The final exam is next Tuesday. Today you review what a model is and how it produces text, and you write the first part of your study guide.
- Board Question: A chatbot writes a confident summary of a book that does not exist. Using what you know about how a model works, explain how that can happen.
Open the discussion form on the right and answer Section 1 from your first reading.
What a Model Is: A Review
This section reviews how a model differs from a program, what lossy compression gives up, and how a language model produces text.
Programs and Training
A program is a set of instructions that a person wrote for a computer to follow. The controls of an elevator are one example. In one published elevator design, the designer wrote a rule that when a call button is pressed, the car stops at that floor and its door opens, then closes after three seconds. A person can read every one of those rules.
A model is software whose behavior came from examples instead of from rules a person wrote. Training is the process of showing a model a large number of examples and adjusting its numbers until its outputs match them. For each example, the model makes a prediction, the prediction is compared with the correct answer, and the model's numbers are updated so the next prediction comes closer. The voice-to-text feature in one phone keyboard is a model of this kind, because its speech recognizer is a model that was trained. Writing a rule by hand is not training, because the rule comes from a person and not from examples. Both run on a chip, so the test is where the behavior came from.
What Lossy Compression Gives Up
Compression makes a file smaller by storing its information in fewer bytes. Lossless compression lets the decoded file match the original exactly. A ZIP file is not lossy, because every part of the original comes back. In a check run for an October lesson, a 36,944-byte text file was zipped to 11,345 bytes and unzipped with 0 characters different.
Lossy compression is compression that removes some detail, so the decoded file is only close to the original and the removed detail cannot be brought back. A podcast episode downloaded as a small file is a common example, since Apple Podcasts accepts episodes as MP3 or AAC files. An MP3 encoder discards sounds that human hearing is less likely to notice, or stores them with less precision. Those sounds are no longer in the file, so no player can rebuild them.
File sizes are measured in units built from the byte, which is 8 bits. From smallest to largest, a kilobyte (KB) is 1,000 bytes, a megabyte (MB) is 1,000,000 bytes, a gigabyte (GB) is 1,000,000,000 bytes, and a terabyte (TB) is 1,000 GB. A 30-minute episode saved at 128 kilobits per second takes 28.8 MB, so a phone with 128 GB could hold about 4,444 such episodes if it stored nothing else.
How a Language Model Produces Text
Andrej Karpathy's 2023 talk "Intro to Large Language Models" describes a language model stored on a drive as two files. The large file is the parameter file, and it holds numbers instead of sentences. For Llama 2 70B, it holds 70 billion parameters at 2 bytes each, or 140 GB. The small file is the runtime, the program that reads those numbers and does the arithmetic. The parameters were set by training on about 10 TB of text, and one useful way to think about the result is as lossy compression of that text. This is only a comparison, because the parameter file stores numbers and not a smaller copy of the text.
A model reads and writes text as tokens, which are words, parts of words or single characters. Prediction is the process of producing a likely next token from the text so far. First, the runtime uses the parameters to calculate how likely each possible next token is. Next, it picks one token and adds it to the end of the text. Then it repeats the loop until the answer is finished. Finding a stored answer and handing it over is not prediction, because nothing new is produced. Figure 1 places the three October figures side by side.
Generating and Retrieving
A search engine retrieves information, which means it finds something stored earlier and returns it, as Google does when it searches its index of web pages. A language model generates text, which means it produces new text by prediction, one token at a time. Karpathy's talk showed a generated store page whose title, author and ISBN number were all made up by the model. A summary of a book that does not exist comes about the same way, because prediction produces text in the form of a summary whether or not any real book matches the title. A confident tone is not evidence, because the same process produces correct and false details in the same fluent style.
Summary
A program follows rules a person wrote, while a model's behavior came from training. Lossless compression gives back every part of a file, while lossy compression removes detail for good. A language model generates text by prediction, one token at a time, even when no real answer exists.
Sources: Google, What is Machine Learning? and Supervised Learning (n.d.); Wen, The Five Floor Elevator Control System Design Based on S7-300 PLC (2016); Schalkwyk, An All-Neural On-Device Speech Recognizer (2019); Deutsch, RFC 1951 (1996); PKWARE, APPNOTE.TXT (2022); CCITT, T.81 (1992); Library of Congress, MP3 (MPEG Layer III Audio Encoding) (2024); Apple, Audio requirements (n.d.); NIST, The Binary Prefixes (n.d.); Karpathy, Intro to Large Language Models (2023); Meta, Llama 2 Model Card (2023); Google, Introduction to Large Language Models (n.d.); Google, In-depth guide to how Google Search works (2025); Kalai et al., Why Language Models Hallucinate (2025); classroom data, ZIP round trip (2026).
-
notebook cue: training
An elevator's controls and a phone's voice-to-text both run on a chip. Which one came from training, and how can you tell?
Check
The voice-to-text came from training, because its speech recognizer is a model that was trained on examples. The elevator's controls follow rules the designer wrote and a person can read, such as stopping at a floor when its call button is pressed. Both run on a chip, so the test is where the behavior came from. See Programs and Training, paragraphs 1 and 2.
-
notebook cue: lossy
A podcast episode is downloaded as a small file. What was removed, and can it be recovered?
Check
The encoder removed sounds that human hearing is less likely to notice, or stored them with less precision. That is lossy compression, so the removed detail is no longer in the file and no player can rebuild it. See What Lossy Compression Gives Up, paragraph 2.
-
notebook cue: prediction
A model has no stored list of answers. What does it use to produce each token?
Check
It uses the numbers in its parameter file. The runtime uses those parameters to calculate how likely each possible next token is, picks one, adds it to the text and repeats. See How a Language Model Produces Text, paragraphs 1 and 2, and the loop on the right of Figure 1.
-
notebook cue: training
An elevator's controls and a phone's voice-to-text both run on a chip. Which one came from training, and how can you tell?
Check
The voice-to-text came from training, because its speech recognizer is a model that was trained on examples. The elevator's controls follow rules the designer wrote and a person can read, such as stopping at a floor when its call button is pressed. Both run on a chip, so the test is where the behavior came from. See Programs and Training, paragraphs 1 and 2.
-
notebook cue: lossy
A podcast episode is downloaded as a small file. What was removed, and can it be recovered?
Check
The encoder removed sounds that human hearing is less likely to notice, or stored them with less precision. That is lossy compression, so the removed detail is no longer in the file and no player can rebuild it. See What Lossy Compression Gives Up, paragraph 2.
-
notebook cue: prediction
A model has no stored list of answers. What does it use to produce each token?
Check
It uses the numbers in its parameter file. The runtime uses those parameters to calculate how likely each possible next token is, picks one, adds it to the text and repeats. See How a Language Model Produces Text, paragraphs 1 and 2, and the loop on the right of Figure 1.
What a Model Is: A Review
This section reviews how a model differs from a program, what lossy compression gives up, and how a language model produces text.
Programs and Training
A program is a set of instructions that a person wrote for a computer to follow. The controls of an elevator are one example. In one published elevator design, the designer wrote a rule that when a call button is pressed, the car stops at that floor and its door opens, then closes after three seconds. A person can read every one of those rules.
A model is software whose behavior came from examples instead of from rules a person wrote. Training is the process of showing a model a large number of examples and adjusting its numbers until its outputs match them. For each example, the model makes a prediction, the prediction is compared with the correct answer, and the model's numbers are updated so the next prediction comes closer. The voice-to-text feature in one phone keyboard is a model of this kind, because its speech recognizer is a model that was trained. Writing a rule by hand is not training, because the rule comes from a person and not from examples. Both run on a chip, so the test is where the behavior came from.
What Lossy Compression Gives Up
Compression makes a file smaller by storing its information in fewer bytes. Lossless compression lets the decoded file match the original exactly. A ZIP file is not lossy, because every part of the original comes back. In a check run for an October lesson, a 36,944-byte text file was zipped to 11,345 bytes and unzipped with 0 characters different.
Lossy compression is compression that removes some detail, so the decoded file is only close to the original and the removed detail cannot be brought back. A podcast episode downloaded as a small file is a common example, since Apple Podcasts accepts episodes as MP3 or AAC files. An MP3 encoder discards sounds that human hearing is less likely to notice, or stores them with less precision. Those sounds are no longer in the file, so no player can rebuild them.
File sizes are measured in units built from the byte, which is 8 bits. From smallest to largest, a kilobyte (KB) is 1,000 bytes, a megabyte (MB) is 1,000,000 bytes, a gigabyte (GB) is 1,000,000,000 bytes, and a terabyte (TB) is 1,000 GB. A 30-minute episode saved at 128 kilobits per second takes 28.8 MB, so a phone with 128 GB could hold about 4,444 such episodes if it stored nothing else.
How a Language Model Produces Text
Andrej Karpathy's 2023 talk "Intro to Large Language Models" describes a language model stored on a drive as two files. The large file is the parameter file, and it holds numbers instead of sentences. For Llama 2 70B, it holds 70 billion parameters at 2 bytes each, or 140 GB. The small file is the runtime, the program that reads those numbers and does the arithmetic. The parameters were set by training on about 10 TB of text, and one useful way to think about the result is as lossy compression of that text. This is only a comparison, because the parameter file stores numbers and not a smaller copy of the text.
A model reads and writes text as tokens, which are words, parts of words or single characters. Prediction is the process of producing a likely next token from the text so far. First, the runtime uses the parameters to calculate how likely each possible next token is. Next, it picks one token and adds it to the end of the text. Then it repeats the loop until the answer is finished. Finding a stored answer and handing it over is not prediction, because nothing new is produced. Figure 1 places the three October figures side by side.
Generating and Retrieving
A search engine retrieves information, which means it finds something stored earlier and returns it, as Google does when it searches its index of web pages. A language model generates text, which means it produces new text by prediction, one token at a time. Karpathy's talk showed a generated store page whose title, author and ISBN number were all made up by the model. A summary of a book that does not exist comes about the same way, because prediction produces text in the form of a summary whether or not any real book matches the title. A confident tone is not evidence, because the same process produces correct and false details in the same fluent style.
Summary
A program follows rules a person wrote, while a model's behavior came from training. Lossless compression gives back every part of a file, while lossy compression removes detail for good. A language model generates text by prediction, one token at a time, even when no real answer exists.
Sources: Google, What is Machine Learning? and Supervised Learning (n.d.); Wen, The Five Floor Elevator Control System Design Based on S7-300 PLC (2016); Schalkwyk, An All-Neural On-Device Speech Recognizer (2019); Deutsch, RFC 1951 (1996); PKWARE, APPNOTE.TXT (2022); CCITT, T.81 (1992); Library of Congress, MP3 (MPEG Layer III Audio Encoding) (2024); Apple, Audio requirements (n.d.); NIST, The Binary Prefixes (n.d.); Karpathy, Intro to Large Language Models (2023); Meta, Llama 2 Model Card (2023); Google, Introduction to Large Language Models (n.d.); Google, In-depth guide to how Google Search works (2025); Kalai et al., Why Language Models Hallucinate (2025); classroom data, ZIP round trip (2026).
Each table agrees on a three-sentence explanation that uses at least two terms from October.
Chromebooks stay closed while your table talks. Sections 1 and 2 of the discussion form are already done. After you talk, finish Section 3 right here under My View. Submit before the exit ticket.
This form has three sections. Section 1: answer today's Board Question. Section 2: after the reading. Section 3 opens later in class. Today's Board Question: A chatbot writes a confident summary of a book that does not exist. Using what you know about how a model works, explain how that can happen.
CR 1. Board Question: A chatbot writes a confident summary of a book that does not exist. Using what you know about how a model works, explain how that can happen. Give your answer and one reason.
Write your answer in your lab notebook, labeled CR 1.
MC 1. Think about your answer to the board question. A chatbot writes a confident summary of a book that does not exist. Using what you know about how a model works, explain how that can happen. Which position is closest to yours now?
- The chatbot found a different book with a similar title and summarized that one.
- The model generated text by prediction, and this time the prediction did not match the facts.
- The chatbot is broken and needs to be repaired.
MC 2. (Reading claim 1) A model keeps a copy of every page it was trained on.
- Agree
- Disagree
MC 3. (Reading claim 2) Lossy compression makes a file smaller by removing some detail.
- Agree
- Disagree
MC 4. (Reading claim 3) A model produces text one token at a time.
- Agree
- Disagree
MC 5. (Reading claim 4) The parameter file for Llama 2 70B takes about 140 GB.
- Agree
- Disagree
MC 6. (Reading claim 5) A ZIP file loses a little detail each time it is unzipped.
- Agree
- Disagree
MC 7. (Reading claim 6) In Figure 1, each new token goes back into the runtime before the next token is picked.
- Agree
- Disagree
MC 8. (Reading claim 7) A search engine writes a new web page to answer each query.
- Agree
- Disagree
MC 9. (Reading claim 8) In Reading 2, Code Coach's explanation of the four-line snippet matched the real output, 6.0.
- Agree
- Disagree
CR 2. (A different view) Write the view at your table that was most different from yours. If everyone agreed, write the strongest argument against your table's answer.
Write your answer in your lab notebook, labeled CR 2.
CR 3. Think about the Board Question again: A chatbot writes a confident summary of a book that does not exist. Using what you know about how a model works, explain how that can happen. Do you have the same opinion as when you first answered the board question? If it is the same, what from the reading or table talk reaffirmed your opinion? If it is different, what from the reading or table talk changed your opinion?
Write your answer in your lab notebook, labeled CR 3.