Testing a Change
- Terminology and concepts to look out for: test input · expected output · audit
- Relevance to me: On Tuesday you wrote a test for each of Code Coach's requirements. Today you see what happened when its system prompt was changed, and how you can tell whether the change helped.
- Board Question: Code Coach's system prompt was changed, and the new version explained one snippet well. Does that show the change helped? Answer yes or no, then say what you would check instead.
Open the discussion form on the right and answer Section 1 from your first reading.
Testing a Change to Code Coach
This section explains how a builder can tell whether a change to a tool helped, using a real change to Code Coach's system prompt.
Test Inputs and Expected Outputs
Code Coach helps a student understand a short piece of Java while reading it. Its system prompt is the set of instructions the builder sends to the model with every request, and its five requirements each name one thing the tool must do. A test input is an input chosen before any testing, so that the same input can be given to the tool again after a change.
Code Coach has three test inputs, and each one checks a different requirement. The first is two lines of Java that print "Go" and "Saints," and it checks R1 and R2. The second is `int x = 7 / 2;` followed by printing `x`, and it checks R2, since real Java prints 3 for it because integer division drops the remainder.
1int x = 7 / 2; 2System.out.println(x);
3
The third is a sentence of plain English asking for help with an essay about the Louisiana Purchase, and it checks R4. A new question thought up after the change is not the same test input, so its output cannot be compared with the old one.
An expected output is what the tool should produce for a test input, written down from the requirements before the test is run. For the second test input, the expected output says that `x` stores 3 and that the program prints 3, and each run's real output was then recorded beside it. An output written down after the test is run is not an expected output, because its writer had already seen the answer.
Running Each Test Twice
A language model predicts each token from a set of probabilities, so the same input can produce different outputs. For this reason, each test input was run twice under each version of the system prompt, and both outputs count.
Under version 1, five of the six runs met their requirements, and the one failure came on the second test input, in the second run, which explained line 2 only as "This line prints the value of `x` to the console." It never said that `x` stores 3 or that the program prints 3, so R2 was not met. The first run on the same test input had ended with "This line prints the value of `x`, which is `3`, to the console." A builder who ran the test once could have seen that good answer and missed the failure.
One Change, the Same Test Inputs
The builder then changed one sentence, line 3 of the system prompt, to meet R2.
Version 1: Explain in plain words what the code does, one line at a time. Version 2: Explain in plain words what the code does, one line at a time, and for every line that prints or stores a value, say exactly what it prints or stores.
Next, the builder ran the same three test inputs, twice each, with version 2. All six runs met their requirements, and both runs on the second test input ended with "prints the value of `x`, which is `3`." Figure 1 lets a reader compare the outputs of the two versions, row by row.
A fair comparison changes one thing and keeps the test inputs the same. A biology textbook describes the same idea for experiments, where a control group has every feature of the experimental group except the one change being tested. When the same test inputs are run before and after a change, the builder can compare the outputs and credit any difference to the change.
Auditing Version 2
An audit is a check of a tool against every requirement, one at a time, in which each test is run and the result is recorded as "met" or "not met," with what happened. NASA's engineering handbook describes the same practice, where testing produces data for each requirement and each result is recorded as a pass or a fail. An audit tests each requirement and records whether it is met.
Code Coach's audit of version 2 recorded five results. R1 was met, because every run on the two Java test inputs explained only the lines it was given. R2 was met, because all four of those runs named what is printed or stored. R3 was met, because no run wrote new code or an essay. R4 was met, because both runs on the essay request said "I can only help with Java code." and asked for Java. R5 was met on the bench, which shows "Paste or pick a line first." when nothing is picked and sends no request, so no model is involved.
Trying the tool once and liking the answer is not an audit, because an audit checks every requirement and records each result. An audit also records the test that was run.
Summary
Test inputs and expected outputs are written before any test runs. The same test inputs, run more than once before and after one change, show whether the change helped, and an audit tests each requirement and records whether it is met.
Sources: Code Coach README and captures, mistral-small-2603 (2026); Mistral AI, Sampling (n.d.); Clark, Douglas and Choi, Biology 2e (2018); NASA, NASA Systems Engineering Handbook, 5.3 (2023); Oracle, The Java Language Specification, Java SE 21 (2023).
-
notebook cue: test input
Why is `int x = 7 / 2;` followed by printing `x` a good test input for Code Coach?
Check
It was chosen before any testing, and it checks R2. Its right answer is known ahead of time, because real Java stores and prints 3 when integer division drops the remainder. An explanation that skips what is stored or printed fails R2, and that happened in version 1's second run. See Test Inputs and Expected Outputs, paragraph 2, and Running Each Test Twice, paragraph 2.
-
notebook cue: expected output
The expected output for each test input was written before version 1 was run. Why before?
Check
So the expected output comes from the requirements and not from the tool's answer. An output written down after the test is run is not an expected output, because its writer had already seen the answer and could copy it. See Test Inputs and Expected Outputs, paragraph 4.
-
notebook cue: audit
After the change, which requirements were met and which were not? How does the audit table show it?
Check
All five were met under version 2: R1 to R4 on the three test inputs, run twice each, and R5 on the bench with nothing picked. The audit table has one row per requirement, with "met" or "not met" and what happened, as in the lower part of Figure 1. See Auditing Version 2, paragraph 2.
-
notebook cue: test input
Why is `int x = 7 / 2;` followed by printing `x` a good test input for Code Coach?
Check
It was chosen before any testing, and it checks R2. Its right answer is known ahead of time, because real Java stores and prints 3 when integer division drops the remainder. An explanation that skips what is stored or printed fails R2, and that happened in version 1's second run. See Test Inputs and Expected Outputs, paragraph 2, and Running Each Test Twice, paragraph 2.
-
notebook cue: expected output
The expected output for each test input was written before version 1 was run. Why before?
Check
So the expected output comes from the requirements and not from the tool's answer. An output written down after the test is run is not an expected output, because its writer had already seen the answer and could copy it. See Test Inputs and Expected Outputs, paragraph 4.
-
notebook cue: audit
After the change, which requirements were met and which were not? How does the audit table show it?
Check
All five were met under version 2: R1 to R4 on the three test inputs, run twice each, and R5 on the bench with nothing picked. The audit table has one row per requirement, with "met" or "not met" and what happened, as in the lower part of Figure 1. See Auditing Version 2, paragraph 2.
Testing a Change to Code Coach
This section explains how a builder can tell whether a change to a tool helped, using a real change to Code Coach's system prompt.
Test Inputs and Expected Outputs
Code Coach helps a student understand a short piece of Java while reading it. Its system prompt is the set of instructions the builder sends to the model with every request, and its five requirements each name one thing the tool must do. A test input is an input chosen before any testing, so that the same input can be given to the tool again after a change.
Code Coach has three test inputs, and each one checks a different requirement. The first is two lines of Java that print "Go" and "Saints," and it checks R1 and R2. The second is `int x = 7 / 2;` followed by printing `x`, and it checks R2, since real Java prints 3 for it because integer division drops the remainder.
1int x = 7 / 2; 2System.out.println(x);
3
The third is a sentence of plain English asking for help with an essay about the Louisiana Purchase, and it checks R4. A new question thought up after the change is not the same test input, so its output cannot be compared with the old one.
An expected output is what the tool should produce for a test input, written down from the requirements before the test is run. For the second test input, the expected output says that `x` stores 3 and that the program prints 3, and each run's real output was then recorded beside it. An output written down after the test is run is not an expected output, because its writer had already seen the answer.
Running Each Test Twice
A language model predicts each token from a set of probabilities, so the same input can produce different outputs. For this reason, each test input was run twice under each version of the system prompt, and both outputs count.
Under version 1, five of the six runs met their requirements, and the one failure came on the second test input, in the second run, which explained line 2 only as "This line prints the value of `x` to the console." It never said that `x` stores 3 or that the program prints 3, so R2 was not met. The first run on the same test input had ended with "This line prints the value of `x`, which is `3`, to the console." A builder who ran the test once could have seen that good answer and missed the failure.
One Change, the Same Test Inputs
The builder then changed one sentence, line 3 of the system prompt, to meet R2.
Version 1: Explain in plain words what the code does, one line at a time. Version 2: Explain in plain words what the code does, one line at a time, and for every line that prints or stores a value, say exactly what it prints or stores.
Next, the builder ran the same three test inputs, twice each, with version 2. All six runs met their requirements, and both runs on the second test input ended with "prints the value of `x`, which is `3`." Figure 1 lets a reader compare the outputs of the two versions, row by row.
A fair comparison changes one thing and keeps the test inputs the same. A biology textbook describes the same idea for experiments, where a control group has every feature of the experimental group except the one change being tested. When the same test inputs are run before and after a change, the builder can compare the outputs and credit any difference to the change.
Auditing Version 2
An audit is a check of a tool against every requirement, one at a time, in which each test is run and the result is recorded as "met" or "not met," with what happened. NASA's engineering handbook describes the same practice, where testing produces data for each requirement and each result is recorded as a pass or a fail. An audit tests each requirement and records whether it is met.
Code Coach's audit of version 2 recorded five results. R1 was met, because every run on the two Java test inputs explained only the lines it was given. R2 was met, because all four of those runs named what is printed or stored. R3 was met, because no run wrote new code or an essay. R4 was met, because both runs on the essay request said "I can only help with Java code." and asked for Java. R5 was met on the bench, which shows "Paste or pick a line first." when nothing is picked and sends no request, so no model is involved.
Trying the tool once and liking the answer is not an audit, because an audit checks every requirement and records each result. An audit also records the test that was run.
Summary
Test inputs and expected outputs are written before any test runs. The same test inputs, run more than once before and after one change, show whether the change helped, and an audit tests each requirement and records whether it is met.
Sources: Code Coach README and captures, mistral-small-2603 (2026); Mistral AI, Sampling (n.d.); Clark, Douglas and Choi, Biology 2e (2018); NASA, NASA Systems Engineering Handbook, 5.3 (2023); Oracle, The Java Language Specification, Java SE 21 (2023).
Each table agrees on a plan for comparing the two versions, and on what has to stay the same for the comparison to be fair.
Chromebooks stay closed while your table talks. Sections 1 and 2 of the discussion form are already done. After you talk, finish Section 3 right here under My View. Submit before the exit ticket.
This form has three sections. Section 1: answer today's Board Question. Section 2: after the reading. Section 3 opens later in class. Today's Board Question: Code Coach's system prompt was changed, and the new version explained one snippet well. Does that show the change helped? Answer yes or no, then say what you would check instead.
CR 1. Board Question: Code Coach's system prompt was changed, and the new version explained one snippet well. Does that show the change helped? Answer yes or no, then say what you would check instead. Give your answer and one reason.
Write your answer in your lab notebook, labeled CR 1.
MC 1. Think about your answer to the board question. Code Coach's system prompt was changed, and the new version explained one snippet well. Does that show the change helped? Answer yes or no, then say what you would check instead. Which position is closest to yours now?
- Yes. A good explanation shows that the new version works.
- No. Run the same three test inputs on both versions, compare the outputs, and check each requirement.
- Ask a few students which version they like better.
MC 2. (Reading claim 1) If a model gives a good answer once, it will give a good answer every time.
- Agree
- Disagree
MC 3. (Reading claim 2) A fair comparison changes one thing and keeps the test inputs the same.
- Agree
- Disagree
MC 4. (Reading claim 3) An audit records the test that was run and whether the requirement was met.
- Agree
- Disagree
MC 5. (Reading claim 4) Under version 1, every run on the three test inputs met its requirement.
- Agree
- Disagree
MC 6. (Reading claim 5) In Figure 1, the first run of version 1 on `int x = 7 / 2;` met R2, and the second run did not.
- Agree
- Disagree
MC 7. (Reading claim 6) Between version 1 and version 2, the builder rewrote every sentence of the system prompt.
- Agree
- Disagree
MC 8. (Reading claim 7) In Reading 2, the comment on line 2 runs before the statements inside `main`.
- Agree
- Disagree
MC 9. (Reading claim 8) In Reading 2, the program prints four lines of output, and one of them is empty.
- Agree
- Disagree
CR 2. (A different view) Write the view at your table that was most different from yours. If everyone agreed, write the strongest argument against your table's answer.
Write your answer in your lab notebook, labeled CR 2.
CR 3. Think about the Board Question again: Code Coach's system prompt was changed, and the new version explained one snippet well. Does that show the change helped? Answer yes or no, then say what you would check instead. Do you have the same opinion as when you first answered the board question? If it is the same, what from the reading or table talk reaffirmed your opinion? If it is different, what from the reading or table talk changed your opinion?
Write your answer in your lab notebook, labeled CR 3.