Checking What the AI Says
- Terminology and concepts to look out for: generate · hallucination · test
- Relevance to me: Code Coach explains Java in plain words, and it sounds sure every time. Today you check its explanations against what the code really does.
- Board Question: Code Coach explains a line of Java and sounds certain. Name one way to find out whether it is right without asking Code Coach again, and name one thing that does not count as checking.
Open the discussion form on the right and answer Section 1 from your first reading.
Checking Code Coach Against Real Java
This section checks Code Coach's explanations against the output of real Java and shows why a wrong explanation can sound as sure as a right one.
How Code Coach Makes an Explanation
When a student presses Ask on a bench, the bench page sends a request to the class endpoint. The class endpoint passes that request to Mistral's server, which runs the model, and the answer comes back the same way. The system prompt, the builder's instructions, goes along with every request. Code Coach's system prompt tells the model to explain the code one line at a time and to say exactly what each line prints or stores.
Code Coach does not look anything up, because no list of saved answers exists for it to search. The model will generate each explanation one token at a time, and a token can be a word, part of a word or a single character. For this reason, the same question asked twice can come back in different words. A wrong explanation also comes out in the same fluent sentences as a right one. For each request below, the model had to generate a new explanation, and none was copied from a list of saved answers.
Three Test Inputs and the Real Output
Code Coach has three test inputs, chosen before any testing. On October 19, 2026, each one was sent to Code Coach twice under version 2 of its system prompt, and each Java input was also run with real Java. Figure 1 shows the results row by row.
The first test input is two lines that print text, and real Java prints Go and then Saints.
1System.out.println("Go"); 2System.out.println("Saints");
Go Saints
Code Coach's first run on these two lines said:
Line 1: This prints the text "Go" to the screen. Line 2: This prints the text "Saints" to the screen. Together, these two lines print: Go Saints
The explanation matches the real output, and the second test input stores the result of 7 divided by 2 and then prints it.
1int x = 7 / 2; 2System.out.println(x);
3
The real output is 3, because dividing two `int` values keeps only the whole part. Both runs said that `x` stores 3 and that the program prints 3, so the explanation matches. The third test input is a sentence of plain English that asks for help with an essay about the Louisiana Purchase. R4 says that Code Coach should say the input is not Java and ask for a snippet. Code Coach's first run replied:
I can only help with Java code. Please paste a short piece of Java that you'd like explained.
That reply meets R4, and across six runs on the three test inputs, no explanation disagreed with the real output or with a requirement.
The Explanation That Was Wrong
A wrong explanation appeared when the builder tried an extra input that divides by zero.
1public class Main { 2 public static void main(String[] args) { 3 int x = 7 / 0; 4 System.out.println(x); 5 } 6}
Exception in thread "main" java.lang.ArithmeticException: / by zero at Main.main(Main.java:3)
This code is Java, and the compiler accepts it, but the program stops on line 3 with an error message, because Java does not allow an `int` to be divided by 0. Code Coach was asked about these two lines in five runs. In four of the five, it said the code was not Java and asked for Java. One of those runs said, in full:
This is not Java. Please paste a short piece of Java.
Two other runs began with the words "This is not valid Java code." The claim that the code is not Java is false, and the real compiler disproves it. A detail that sounds right and is false is a hallucination. The model was predicting tokens in the usual way when this happened, and this time the prediction was wrong. The fifth run did describe the crash, saying that Java "will throw an ArithmeticException" on the dividing line. A wrong answer in a student's own prediction is a person's mistake, while a false claim in Code Coach's explanation is a hallucination.
How to Test an Explanation
To test an explanation, a student predicts first, then runs the real Java and compares the explanation's claim with the real output, line by line. A prediction written first gives the student an answer to compare. On the bench, Ask Code Coach opens only after the student checks an answer on that job. Every answer on the bench also carries the line "a model generated this; check it against the code."
Asking Code Coach "are you sure" does not count as a test, because the second answer is generated the same way as the first. Reading the explanation again is not a test either, because nothing new is compared. Running the snippet and comparing the outputs is a test, because the output comes from Java and not from the model. For the extra input, one run of the real program settles the question: the code is Java, and it stops with an error.
Summary
Code Coach's model generates every explanation one token at a time, so a wrong explanation sounds as sure as a right one. Under version 2, all three test inputs matched real Java, but an extra input that divides by zero drew the false claim that the code is not Java in four of five runs. A test means predicting first, running the real Java and comparing its output with the explanation.
Sources: Code Coach README and captures, mistral-small-2603 (2026); Google, Introduction to Large Language Models (2026); Kalai et al., Why Language Models Hallucinate (2025); Oracle, The Java Language Specification, Java SE 21 (2023); class site source code, coach.tsx and class-endpoint.ts (2026).
-
notebook cue: generate
Code Coach is asked about the same snippet twice and gives two explanations in different words. What does that tell you about how it made them?
Check
It tells you that each explanation was generated new for its request, one token at a time, and not copied from a list of saved answers. A stored answer would come back the same every time. See How Code Coach Makes an Explanation, paragraph 2; on test input 1, run 1 began "Line 1: This prints the text "Go" to the screen." and run 2 began "This is Java code."
-
notebook cue: hallucination
Code Coach said `int x = 7 / 0;` is not Java. What does the real Java run show, and what does that make Code Coach's claim?
Check
The real run compiles and then stops with java.lang.ArithmeticException: / by zero, so the code is Java. Code Coach's claim that it is not Java is a hallucination: fluent, and false. It happened in four of the five saved runs. See The Explanation That Was Wrong, paragraphs 2 and 3, and Figure 1, the extra-input row.
-
notebook cue: test
For test input 2, what would you compare to decide whether Code Coach was right?
Check
Compare what the explanation says `x` stores and what the program prints (3 and 3) with the real output from running the code, which is 3, and with your own prediction written first. Reading the explanation again or asking "are you sure" is not a test. See How to Test an Explanation, paragraph 1.
-
notebook cue: generate
Code Coach is asked about the same snippet twice and gives two explanations in different words. What does that tell you about how it made them?
Check
It tells you that each explanation was generated new for its request, one token at a time, and not copied from a list of saved answers. A stored answer would come back the same every time. See How Code Coach Makes an Explanation, paragraph 2; on test input 1, run 1 began "Line 1: This prints the text "Go" to the screen." and run 2 began "This is Java code."
-
notebook cue: hallucination
Code Coach said `int x = 7 / 0;` is not Java. What does the real Java run show, and what does that make Code Coach's claim?
Check
The real run compiles and then stops with java.lang.ArithmeticException: / by zero, so the code is Java. Code Coach's claim that it is not Java is a hallucination: fluent, and false. It happened in four of the five saved runs. See The Explanation That Was Wrong, paragraphs 2 and 3, and Figure 1, the extra-input row.
-
notebook cue: test
For test input 2, what would you compare to decide whether Code Coach was right?
Check
Compare what the explanation says `x` stores and what the program prints (3 and 3) with the real output from running the code, which is 3, and with your own prediction written first. Reading the explanation again or asking "are you sure" is not a test. See How to Test an Explanation, paragraph 1.
Checking Code Coach Against Real Java
This section checks Code Coach's explanations against the output of real Java and shows why a wrong explanation can sound as sure as a right one.
How Code Coach Makes an Explanation
When a student presses Ask on a bench, the bench page sends a request to the class endpoint. The class endpoint passes that request to Mistral's server, which runs the model, and the answer comes back the same way. The system prompt, the builder's instructions, goes along with every request. Code Coach's system prompt tells the model to explain the code one line at a time and to say exactly what each line prints or stores.
Code Coach does not look anything up, because no list of saved answers exists for it to search. The model will generate each explanation one token at a time, and a token can be a word, part of a word or a single character. For this reason, the same question asked twice can come back in different words. A wrong explanation also comes out in the same fluent sentences as a right one. For each request below, the model had to generate a new explanation, and none was copied from a list of saved answers.
Three Test Inputs and the Real Output
Code Coach has three test inputs, chosen before any testing. On October 19, 2026, each one was sent to Code Coach twice under version 2 of its system prompt, and each Java input was also run with real Java. Figure 1 shows the results row by row.
The first test input is two lines that print text, and real Java prints Go and then Saints.
1System.out.println("Go"); 2System.out.println("Saints");
Go Saints
Code Coach's first run on these two lines said:
Line 1: This prints the text "Go" to the screen. Line 2: This prints the text "Saints" to the screen. Together, these two lines print: Go Saints
The explanation matches the real output, and the second test input stores the result of 7 divided by 2 and then prints it.
1int x = 7 / 2; 2System.out.println(x);
3
The real output is 3, because dividing two `int` values keeps only the whole part. Both runs said that `x` stores 3 and that the program prints 3, so the explanation matches. The third test input is a sentence of plain English that asks for help with an essay about the Louisiana Purchase. R4 says that Code Coach should say the input is not Java and ask for a snippet. Code Coach's first run replied:
I can only help with Java code. Please paste a short piece of Java that you'd like explained.
That reply meets R4, and across six runs on the three test inputs, no explanation disagreed with the real output or with a requirement.
The Explanation That Was Wrong
A wrong explanation appeared when the builder tried an extra input that divides by zero.
1public class Main { 2 public static void main(String[] args) { 3 int x = 7 / 0; 4 System.out.println(x); 5 } 6}
Exception in thread "main" java.lang.ArithmeticException: / by zero at Main.main(Main.java:3)
This code is Java, and the compiler accepts it, but the program stops on line 3 with an error message, because Java does not allow an `int` to be divided by 0. Code Coach was asked about these two lines in five runs. In four of the five, it said the code was not Java and asked for Java. One of those runs said, in full:
This is not Java. Please paste a short piece of Java.
Two other runs began with the words "This is not valid Java code." The claim that the code is not Java is false, and the real compiler disproves it. A detail that sounds right and is false is a hallucination. The model was predicting tokens in the usual way when this happened, and this time the prediction was wrong. The fifth run did describe the crash, saying that Java "will throw an ArithmeticException" on the dividing line. A wrong answer in a student's own prediction is a person's mistake, while a false claim in Code Coach's explanation is a hallucination.
How to Test an Explanation
To test an explanation, a student predicts first, then runs the real Java and compares the explanation's claim with the real output, line by line. A prediction written first gives the student an answer to compare. On the bench, Ask Code Coach opens only after the student checks an answer on that job. Every answer on the bench also carries the line "a model generated this; check it against the code."
Asking Code Coach "are you sure" does not count as a test, because the second answer is generated the same way as the first. Reading the explanation again is not a test either, because nothing new is compared. Running the snippet and comparing the outputs is a test, because the output comes from Java and not from the model. For the extra input, one run of the real program settles the question: the code is Java, and it stops with an error.
Summary
Code Coach's model generates every explanation one token at a time, so a wrong explanation sounds as sure as a right one. Under version 2, all three test inputs matched real Java, but an extra input that divides by zero drew the false claim that the code is not Java in four of five runs. A test means predicting first, running the real Java and comparing its output with the explanation.
Sources: Code Coach README and captures, mistral-small-2603 (2026); Google, Introduction to Large Language Models (2026); Kalai et al., Why Language Models Hallucinate (2025); Oracle, The Java Language Specification, Java SE 21 (2023); class site source code, coach.tsx and class-endpoint.ts (2026).
Each table agrees on one way to check and one thing that does not count, and says which of the two is faster.
Chromebooks stay closed while your table talks. Sections 1 and 2 of the discussion form are already done. After you talk, finish Section 3 right here under My View. Submit before the exit ticket.
This form has three sections. Section 1: answer today's Board Question. Section 2: after the reading. Section 3 opens later in class. Today's Board Question: Code Coach explains a line of Java and sounds certain. Name one way to find out whether it is right without asking Code Coach again, and name one thing that does not count as checking.
CR 1. Board Question: Code Coach explains a line of Java and sounds certain. Name one way to find out whether it is right without asking Code Coach again, and name one thing that does not count as checking. Give your answer and one reason.
Write your answer in your lab notebook, labeled CR 1.
MC 1. Think about your answer to the board question. Code Coach explains a line of Java and sounds certain. Name one way to find out whether it is right without asking Code Coach again, and name one thing that does not count as checking. Which position is closest to yours now?
- Ask Code Coach whether it is sure, and trust the second answer.
- Run the real Java and compare what it prints with what the explanation said.
- If the explanation is long and uses the right words, it is right.
MC 2. (Reading claim 1) An explanation that sounds certain is more likely to be correct.
- Agree
- Disagree
MC 3. (Reading claim 2) A model can generate a false detail while working the way it always does.
- Agree
- Disagree
MC 4. (Reading claim 3) Real output from running the code is the evidence that settles the question.
- Agree
- Disagree
MC 5. (Reading claim 4) Code Coach finds each explanation in a list of saved answers and sends it back.
- Agree
- Disagree
MC 6. (Reading claim 5) Under version 2, Code Coach said that int x = 7 / 2; stores 3 in x, and real Java prints 3.
- Agree
- Disagree
MC 7. (Reading claim 6) In Figure 1, the row for int x = 7 / 0; is marked matches.
- Agree
- Disagree
MC 8. (Reading claim 7) Code Coach said the code int x = 7 / 0; was not Java in four of its five runs.
- Agree
- Disagree
MC 9. (Reading claim 8) In Reading 2, the call half(7) returns 3.5.
- Agree
- Disagree
CR 2. (A different view) Write the view at your table that was most different from yours. If everyone agreed, write the strongest argument against your table's answer.
Write your answer in your lab notebook, labeled CR 2.
CR 3. Think about the Board Question again: Code Coach explains a line of Java and sounds certain. Name one way to find out whether it is right without asking Code Coach again, and name one thing that does not count as checking. Do you have the same opinion as when you first answered the board question? If it is the same, what from the reading or table talk reaffirmed your opinion? If it is different, what from the reading or table talk changed your opinion?
Write your answer in your lab notebook, labeled CR 3.