Before the bell
You are responsible for all of this before the end of class.
- The discussion form in the I Do deck. All three sections. If you are here, this should be done.
- Your classwork. This form on the right. Section 1 before the exit ticket pops up; Section 2 past the floor.
- Today's Bench Basics activity: Find the File. It opens later in class and lives on slide 4 of this deck.
- The exit ticket. It opens 15 minutes before the bell and lives on the last slide of this deck.
Sign out of Google when you finish on a shared computer, so the next student can sign in as themselves. Google then shows a "Redirect notice" page. Click the muggsofcompsci.net/cs1 link on it to come back.
Where the Numbers Came From
This reading explains how training set the numbers in a parameter file, and why a detail from training can come back wrong.
Comparing Two Sizes
Training is the process that sets a model's parameters. During training, a computer processes a very large amount of text and adjusts the parameters so that the model predicts the next token more accurately. Meta's model card states that Llama 2 was trained on 2 trillion tokens of data from publicly available sources. In his 2023 talk, Andrej Karpathy describes the training text for Llama 2 70B as about 10 TB. The parameter file that came out of training is 140 GB. These two sizes can be compared directly, because 10 TB is about 10,000 GB. The text is about 70 times as large as the parameter file. A file that size cannot hold an exact copy of the text, so some detail from the text was not kept. This is why lossy compression is one useful way to think about training, even though training is not the same process as saving a JPEG. A token is the unit a model reads and writes. A token is often a whole word, but it can be a part of a word or a punctuation mark, so the 2 trillion token count depends on the tokenizer that Meta used. The 10 TB figure is the size of the raw text on disk, and the 140 GB figure is the size of the parameter file that training produced from that text. Neither number is the training recipe. Training is a long sequence of small adjustments to the numbers, each one made because a prediction landed a little closer or a little further from the next token in the training text.
Why Details Come Back Wrong
When a model generates text, it predicts each token from patterns stored in its parameters. Some details from the training text are stored closely, and others are not. In the talk, a model generated a page that looked like a book listing, with a title, an author and an ISBN number. Karpathy explains that the ISBN number was almost certainly made up. The model produced a number of about the right length in the place where an ISBN usually appears, but the digits themselves were a prediction and not a stored record. A detail that appeared only once in 10 TB of text is a very small part of what the model was trained on. A person reading the output cannot tell from the text alone which details were stored closely and which were not. This is why a model's answer should be treated as a draft that needs a check against a source. A fact that is likely stored closely, such as a widely-quoted sentence from a very common book, often comes back the same way every time. A fact that is rarely written down, such as a specific ISBN or the date a small company was founded, is more likely to come back as a plausible-looking guess. The reader has no way to tell the two apart from the text alone, which makes checking the first step and not the last.
Sources: Karpathy, "[1hr Talk] Intro to Large Language Models" (2023); Meta, Llama 2 Model Card (2023).
Where the Numbers Came From, and Why Some of Them Wander Off
This entry explains how training set the numbers in a parameter file, and why a detail from training can come back wrong while looking perfectly respectable.
Comparing Two Sizes
Training is the process that sets a model's parameters. During training, a computer works through a very large amount of text and nudges the parameters so that the model predicts the next token more accurately. Meta's model card states that Llama 2 was trained on 2 trillion tokens of data from publicly available sources. In his 2023 talk, Andrej Karpathy describes the training text for Llama 2 70B as about 10 TB. The parameter file that came out of training is 140 GB. These two sizes can be compared directly, because 10 TB is about 10,000 GB. The text is about 70 times as large as the parameter file. A file that size cannot hold an exact copy of the text, so some detail from the text was simply not kept, and nobody left a note saying which. This is why lossy compression is one useful way to think about training, even though training is not the same process as saving a JPEG. A token is the unit a model reads and writes. A token is often a whole word, but it can be part of a word or a punctuation mark, so the 2 trillion token count depends on the tokenizer that Meta used. The 10 TB figure is the size of the raw text on disk, and the 140 GB figure is the size of the parameter file that training produced from that text. Neither number is the training recipe. Training is a long, long sequence of small adjustments to the numbers, each one made because a prediction landed a little closer to, or a little further from, the next token in the training text.
Why Details Come Back Wrong
When a model generates text, it predicts each token from patterns stored in its parameters. Some details from the training text are stored closely, and others are not. In the talk, a model generated a page that looked like a book listing, with a title, an author and an ISBN number. Karpathy explains that the ISBN number was almost certainly made up. The model produced a number of about the right length in the place where an ISBN usually appears, but the digits themselves were a prediction and not a stored record. It knew the shape of the answer, just not the answer. A detail that appeared only once in 10 TB of text is a very small part of what the model was trained on, roughly the size of one grain of sand on a long beach. A person reading the output cannot tell from the text alone which details were stored closely and which were not. This is why a model's answer should be treated as a draft that needs a check against a source. A fact that is likely stored closely, such as a widely-quoted sentence from a very common book, often comes back the same way every time. A fact that is rarely written down, such as a specific ISBN or the date a small company was founded, is more likely to come back as a plausible-looking guess, wearing a nice shirt. The reader has no way to tell the two apart from the text alone, which makes checking the first step and not the last.
Sources: Karpathy, "[1hr Talk] Intro to Large Language Models" (2023); Meta, Llama 2 Model Card (2023).
A Language Model Is Two Files
This section explains what a language model is made of when it is stored on a computer, and what it does each time it runs.
The Parameter File
A language model is a computer program that predicts text. When a language model is stored on a computer, it can be described as just two files. One example is Llama 2 70B, a model that Meta released for anyone to download and run. Parameters are the numbers a language model uses to calculate what text comes next. The larger of the two files holds the parameters. Llama 2 70B has 70 billion parameters. Each parameter is stored as a number that takes up 2 bytes, so the parameter file for this model is 140 GB. A person who opened that file would see a long list of numbers and would not find a single sentence written out in it. A parameter file is not an example of a saved document, because a saved document holds its sentences and gives them back exactly. The parameter file holds no sentences, so its numbers are useful only when a program reads them and does arithmetic with them.
Where the Parameters Come From
The parameters are set by training, a process in which a computer adjusts the numbers while it processes a very large amount of text. For Llama 2 70B, the training data was about 10 TB of text taken from the internet. One useful way to think about the result is as lossy compression of the text that was used in training. Lossy compression makes a file smaller by removing detail that cannot be recovered, like the JPEG photos from last week. The 140 GB parameter file is far smaller than the 10 TB of text, so it cannot hold an exact copy of that text. This is a comparison and not an exact description of what training does. A JPEG still stores a picture, while the parameter file stores only numbers that help predict text.
The Runtime
The second, much smaller file is the runtime. A runtime is the program that reads the parameter file and does the arithmetic that turns those numbers into text. Andrej Karpathy's 2023 talk "Intro to Large Language Models" describes a runtime for Llama 2 70B that takes about 500 lines of code in the C programming language. Another example is llama.cpp, a runtime written in C and C++ that needs no other software libraries to work. A runtime with no parameter file has nothing to run, because the arithmetic needs the numbers that the parameter file stores. A parameter file with no runtime sits on a drive and produces nothing, because no program is reading its numbers. When both files are on the same computer, they form a complete package, and the model can run without any connection to the internet. Mr. Mugg's computer at home runs open models this way, as two files stored on a drive. MuggsOfPrompts and MuggsOfVibes, the tools used in this class, work differently. They send each prompt over the internet to a model that runs on Mistral's servers.
Prediction, One Token at a Time
A model reads and produces text as tokens, which are words or parts of words that a computer represents as numbers. Prediction is the process of producing a likely next token from the text so far. The runtime carries out prediction in a loop of four steps, as Figure 1 shows. First, it reads all the tokens so far, starting with the prompt. Next, it uses the parameters to score every possible next token by calculating how likely each one is. Then it picks one token. Finally, it adds that token to the end of the text and starts the loop again with a list that is one token longer. For example, given the words "cat sat on a," a model might predict "mat" as the next word. The model repeats the loop one token at a time until the answer is complete. Prediction is not the same as a lookup, because a lookup hands back something that was stored, while a prediction produces something new. For this reason, a model can generate a sentence that does not appear anywhere in its training text. That sentence may be correct, or it may contain details that are false.
Summary
A language model on a computer is two files: a large parameter file that stores numbers and a small runtime that does the arithmetic with them. Each time the model runs, it makes a prediction of the next token, adds that token to the text, and repeats. Training on a very large amount of text set the parameters, and lossy compression is one useful comparison for the result, though not an exact description.
Sources: Karpathy, "[1hr Talk] Intro to Large Language Models" (2023); Meta, Llama 2 Model Card (2023); Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023); ggml-org, llama.cpp README (2026); Hugging Face, LLM Course, "Tokenizers" (n.d.); classroom data (2026).
A Language Model Is Two Files, Which Is Fewer Than You Would Think
This entry explains what a language model is actually made of when it sits on a computer, and what it does every time someone wakes it up.
The Parameter File
A language model is a computer program that predicts text. Stored on a computer, it can be described as just two files, which is a little underwhelming for something people argue about this much. One example is Llama 2 70B, a model that Meta released for anyone to download and run. Parameters are the numbers a language model uses to calculate what text comes next. The larger of the two files holds the parameters. Llama 2 70B has 70 billion parameters. Each parameter is stored as a number that takes up 2 bytes, so the parameter file for this model is 140 GB. A person who opened that file would see a very long list of numbers and would not find a single sentence written out anywhere, no matter how long they scrolled. A parameter file is not an example of a saved document, because a saved document holds its sentences and gives them back exactly. The parameter file holds no sentences at all, so its numbers are useful only when a program reads them and does arithmetic with them.
Where the Parameters Come From
The parameters are set by training, a process in which a computer adjusts the numbers while it works through a very large amount of text. For Llama 2 70B, the training data was about 10 TB of text taken from the internet, which is to say a great deal of everything. One useful way to think about the result is as lossy compression of the text used in training. Lossy compression makes a file smaller by removing detail that cannot be recovered, like the JPEG photos from last week. The 140 GB parameter file is far smaller than the 10 TB of text, so it cannot hold an exact copy of that text. Something had to go. This is a comparison and not an exact description of what training does. A JPEG still stores a picture, while the parameter file stores only numbers that help predict text.
The Runtime
The second, much smaller file is the runtime. A runtime is the program that reads the parameter file and does the arithmetic that turns those numbers into text. Andrej Karpathy's 2023 talk "Intro to Large Language Models" describes a runtime for Llama 2 70B that takes about 500 lines of code in the C programming language, which is roughly the length of a long essay nobody asked for. Another example is llama.cpp, a runtime written in C and C++ that needs no other software libraries to work. A runtime with no parameter file has nothing to run, because the arithmetic needs the numbers that the parameter file stores. A parameter file with no runtime just sits on a drive and produces nothing, like a cookbook in a house with no kitchen. When both files are on the same computer, they form a complete package, and the model can run without any connection to the internet. Mr. Mugg's computer at home runs open models this way, as two files stored on a drive. MuggsOfPrompts and MuggsOfVibes, the tools used in this class, work differently. They send each prompt over the internet to a model that runs on Mistral's servers, somewhere else entirely.
Prediction, One Token at a Time
A model reads and produces text as tokens, which are words or parts of words that a computer represents as numbers. Prediction is the process of producing a likely next token from the text so far. The runtime carries out prediction in a loop of four steps, as Figure 1 shows. First, it reads all the tokens so far, starting with the prompt. Next, it uses the parameters to score every possible next token by calculating how likely each one is. Then it picks one token. Finally, it adds that token to the end of the text and starts the loop again with a list that is one token longer. It does not get bored. For example, given the words "cat sat on a," a model might predict "mat" as the next word. The model repeats the loop one token at a time until the answer is complete. Prediction is not the same as a lookup, because a lookup hands back something that was stored, while a prediction produces something new. For this reason, a model can generate a sentence that does not appear anywhere in its training text. That sentence may be correct, or it may contain details that are false, and it will sound exactly the same either way.
Summary
A language model on a computer is two files: a large parameter file that stores numbers and a small runtime that does the arithmetic with them. Each time the model runs, it makes a prediction of the next token, adds that token to the text, and repeats, patiently, forever if necessary. Training on a very large amount of text set the parameters, and lossy compression is one useful comparison for the result, though not an exact description.
Sources: Karpathy, "[1hr Talk] Intro to Large Language Models" (2023); Meta, Llama 2 Model Card (2023); Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023); ggml-org, llama.cpp README (2026); Hugging Face, LLM Course, "Tokenizers" (n.d.); classroom data (2026).
Sign out of Google when you finish on a shared computer, so the next student can sign in as themselves. Google then shows a "Redirect notice" page. Click the muggsofcompsci.net/cs1 link on it to come back.
CR 1. In Figure 1, the large box is labeled parameters (large file). What does the large file do? Use one number from Reading 2 or Figure 1.
Write your answer in your lab notebook, labeled CR 1.
CR 2. In Figure 1, the small box is labeled runtime (small file). What does the small file do?
Write your answer in your lab notebook, labeled CR 2.
MC 1. (Step 1) Figure 1 shows the four steps the runtime repeats on every pass of the loop. Which step comes first, before anything has been scored?
- Read all the tokens so far
- Score every possible next token
- Pick one token
- Add the token to the end of the text
MC 2. (Step 2) The runtime has read the tokens so far. Which step comes next?
- Read all the tokens so far
- Score every possible next token
- Pick one token
- Add the token to the end of the text
MC 3. (Step 3) The runtime has calculated how likely each possible next token is. Which step comes next?
- Read all the tokens so far
- Score every possible next token
- Pick one token
- Add the token to the end of the text
MC 4. (Step 4) The runtime has picked one token. Which step finishes this pass, so the loop can go again?
- Read all the tokens so far
- Score every possible next token
- Pick one token
- Add the token to the end of the text
Sign out of Google when you finish on a shared computer, so the next student can sign in as themselves. Google then shows a "Redirect notice" page. Click the muggsofcompsci.net/cs1 link on it to come back.
CR 3. Last week, a lossy JPEG lost detail you could see, such as sharp edges in a photo. When the file is a parameter file, what kind of detail is lost? Use one size from Reading 2.
Write your answer in your lab notebook, labeled CR 3.
CR 4. When you open a lossy JPEG, you get the same photo back with less detail. When a model gives back a detail from its training text, it can be a new detail that is wrong. Explain the difference. Use the word prediction in your answer.
Write your answer in your lab notebook, labeled CR 4.
Find the File
Open the bench, do its six jobs, then paste your credit code into the credit-code form. You can turn in up to three codes per bench; your best score is your grade.
Bench Basics opens during your class period.
Sign out of Google when you finish on a shared computer, so the next student can sign in as themselves. Google then shows a "Redirect notice" page. Click the muggsofcompsci.net/cs1 link on it to come back.
Sign out of Google when you finish on a shared computer, so the next student can sign in as themselves. Google then shows a "Redirect notice" page. Click the muggsofcompsci.net/cs1 link on it to come back.
This opens during your class period.