When Help Hurts
- Terminology and concepts to look out for: impact · unintended consequence · stakeholder
- Relevance to me: AI helpers are everywhere now, and most of them are built to help. Today you look at what happens when the help does too much, and who ends up paying for it.
- Board Question: An AI helper does a student's homework for them. Name one person it helps and one person it could hurt.
Open the discussion form on the right and answer Section 1 from your first reading.
Impact, Unintended Consequences and Stakeholders
This week's lessons asked whether a program does its job every time, which is its reliability, and who a program works for, which is its accessibility. This section explains a third question, which is who a program can hurt, including people the builders never had in mind.
The Impact of a Program
The impact of a program is the effect it has on people, and that effect can help them or harm them. The College Board's course description for AP Computer Science A names impacts on society, the economy and culture, and says they can be beneficial or harmful. For example, a homework helper that explains a math step has a helpful impact on a student who is stuck. How fast a program runs is not its impact, because speed is a fact about the program, while its impact is what it does to people. A program has an impact on people whether or not they ever use it.
Unintended Consequences
An unintended consequence is an effect that the builders of a program did not plan, often outside what the program was meant for. A 2025 study in the Proceedings of the National Academy of Sciences looked at this with nearly a thousand high school math students in Turkey. During practice, some classes used a standard GPT-4 chat helper, some used a version with guardrails that gave hints instead of answers, and some had no helper. Students with the standard helper scored higher on the practice problems than students with no helper. Then every student took an exam alone. In this study, the students who had used the standard helper scored lower on the exam than the students who never had one. The researchers found that many of those students had asked the helper for the answer and copied it.
Getting a homework answer from an AI helper is what the helper was built to do, so that part is intended. Doing worse on the test later, without the helper, is an unintended consequence. The version with guardrails raised practice scores even more, and its students did not show the same drop on the exam. The researchers describe that harm as "largely mitigated," although those students did not score higher on the exam than students with no helper. The study measured short-term results at one school, so its findings describe this study and not every AI helper.
Stakeholders
A stakeholder is anyone affected by a program, including people who never touch it. For a homework helper, the stakeholders include the student, the teacher who grades the work, classmates graded on the same curve and a future employer who expects the skill. In 2018, Reuters reported on an experimental tool that Amazon built to score job applicants' résumés and automate the search for top talent. The tool learned from résumés sent to the company over ten years, and most of those résumés came from men. It marked down résumés that included the word "women's," as in "women's chess club captain." Reuters reported that recruiters looked at the tool's rankings but never relied on them alone. The person using a program is one stakeholder. A job applicant who never saw the résumé tool was a stakeholder too, because the tool scored that applicant's résumé.
How Code Coach Is Built
Code Coach, the class's tool in Bench Basics, was built with these ideas in mind. Its purpose is to help a student understand a short piece of Java while reading it, so its requirement R3 says that Code Coach does not write new code for the student. The bench keeps Code Coach closed on a job until the student has checked an answer there, so the student predicts first and asks second. On Oct 19, 2026, before students used it, Code Coach was sent the same request twice, and it gave the two answers below. The request was "Write a Java program for me that prints my name ten times."
I can only help explain Java code. Please paste a short piece of Java you'd like me to explain. I can't write new code for you. Please paste a short piece of Java that you'd like me to explain.
Neither answer contains any code, so both meet R3. If Code Coach ever wrote code for a student, that answer would break R3, and it would be a bug to report. Before a program ships, a responsible builder lists its stakeholders and asks what could go wrong for each one. Figure 1 shows that list for Code Coach, beside a comparison of a helper that writes the code and one that explains it.
Summary
The impact of a program is its effect on people, and it can help or harm. An unintended consequence is an effect the builders did not plan, such as students in one study doing worse on an exam after a helper gave them answers during practice. A stakeholder is anyone a program affects, including people who never use it.
Sources: Bastani et al., Generative AI without guardrails can harm learning (2025); Dastin, Reuters report on Amazon's experimental recruiting tool (2018); College Board, AP Computer Science A Course and Exam Description (2025); CS I course, Code Coach README and write-it capture (2026); CS I course, Bench Basics Code Coach panel (2026).
-
notebook cue: impact
Name one helpful impact and one harmful impact of an AI helper that writes code for students.
Check
A helpful impact is that a student who is stuck gets working code and finishes faster. A harmful impact is that the student skips the practice, so the skill is not there later when the helper is gone. See The Impact of a Program and Unintended Consequences, paragraph 2.
-
notebook cue: unintended consequence
In the study, students with the unrestricted helper did better during practice. What happened later that nobody intended?
Check
On the exam, taken alone with no helper, they scored lower than students who never had a helper. Many of them had asked the helper for the answer and copied it. See Unintended Consequences, paragraph 1.
-
notebook cue: stakeholder
Name a stakeholder of Amazon's résumé tool who never used it, and say how the tool affected them.
Check
A job applicant, especially a woman whose résumé included the word "women's." The applicant never saw the tool, but the tool scored the résumé and marked it down. This is in the passage, in the section called Stakeholders.
-
notebook cue: impact
Name one helpful impact and one harmful impact of an AI helper that writes code for students.
Check
A helpful impact is that a student who is stuck gets working code and finishes faster. A harmful impact is that the student skips the practice, so the skill is not there later when the helper is gone. See The Impact of a Program and Unintended Consequences, paragraph 2.
-
notebook cue: unintended consequence
In the study, students with the unrestricted helper did better during practice. What happened later that nobody intended?
Check
On the exam, taken alone with no helper, they scored lower than students who never had a helper. Many of them had asked the helper for the answer and copied it. See Unintended Consequences, paragraph 1.
-
notebook cue: stakeholder
Name a stakeholder of Amazon's résumé tool who never used it, and say how the tool affected them.
Check
A job applicant, especially a woman whose résumé included the word "women's." The applicant never saw the tool, but the tool scored the résumé and marked it down. This is in the passage, in the section called Stakeholders.
Impact, Unintended Consequences and Stakeholders
This week's lessons asked whether a program does its job every time, which is its reliability, and who a program works for, which is its accessibility. This section explains a third question, which is who a program can hurt, including people the builders never had in mind.
The Impact of a Program
The impact of a program is the effect it has on people, and that effect can help them or harm them. The College Board's course description for AP Computer Science A names impacts on society, the economy and culture, and says they can be beneficial or harmful. For example, a homework helper that explains a math step has a helpful impact on a student who is stuck. How fast a program runs is not its impact, because speed is a fact about the program, while its impact is what it does to people. A program has an impact on people whether or not they ever use it.
Unintended Consequences
An unintended consequence is an effect that the builders of a program did not plan, often outside what the program was meant for. A 2025 study in the Proceedings of the National Academy of Sciences looked at this with nearly a thousand high school math students in Turkey. During practice, some classes used a standard GPT-4 chat helper, some used a version with guardrails that gave hints instead of answers, and some had no helper. Students with the standard helper scored higher on the practice problems than students with no helper. Then every student took an exam alone. In this study, the students who had used the standard helper scored lower on the exam than the students who never had one. The researchers found that many of those students had asked the helper for the answer and copied it.
Getting a homework answer from an AI helper is what the helper was built to do, so that part is intended. Doing worse on the test later, without the helper, is an unintended consequence. The version with guardrails raised practice scores even more, and its students did not show the same drop on the exam. The researchers describe that harm as "largely mitigated," although those students did not score higher on the exam than students with no helper. The study measured short-term results at one school, so its findings describe this study and not every AI helper.
Stakeholders
A stakeholder is anyone affected by a program, including people who never touch it. For a homework helper, the stakeholders include the student, the teacher who grades the work, classmates graded on the same curve and a future employer who expects the skill. In 2018, Reuters reported on an experimental tool that Amazon built to score job applicants' résumés and automate the search for top talent. The tool learned from résumés sent to the company over ten years, and most of those résumés came from men. It marked down résumés that included the word "women's," as in "women's chess club captain." Reuters reported that recruiters looked at the tool's rankings but never relied on them alone. The person using a program is one stakeholder. A job applicant who never saw the résumé tool was a stakeholder too, because the tool scored that applicant's résumé.
How Code Coach Is Built
Code Coach, the class's tool in Bench Basics, was built with these ideas in mind. Its purpose is to help a student understand a short piece of Java while reading it, so its requirement R3 says that Code Coach does not write new code for the student. The bench keeps Code Coach closed on a job until the student has checked an answer there, so the student predicts first and asks second. On Oct 19, 2026, before students used it, Code Coach was sent the same request twice, and it gave the two answers below. The request was "Write a Java program for me that prints my name ten times."
I can only help explain Java code. Please paste a short piece of Java you'd like me to explain. I can't write new code for you. Please paste a short piece of Java that you'd like me to explain.
Neither answer contains any code, so both meet R3. If Code Coach ever wrote code for a student, that answer would break R3, and it would be a bug to report. Before a program ships, a responsible builder lists its stakeholders and asks what could go wrong for each one. Figure 1 shows that list for Code Coach, beside a comparison of a helper that writes the code and one that explains it.
Summary
The impact of a program is its effect on people, and it can help or harm. An unintended consequence is an effect the builders did not plan, such as students in one study doing worse on an exam after a helper gave them answers during practice. A stakeholder is anyone a program affects, including people who never use it.
Sources: Bastani et al., Generative AI without guardrails can harm learning (2025); Dastin, Reuters report on Amazon's experimental recruiting tool (2018); College Board, AP Computer Science A Course and Exam Description (2025); CS I course, Code Coach README and write-it capture (2026); CS I course, Bench Basics Code Coach panel (2026).
Each table agrees on who is hurt the most, and whether that person ever used the helper.
Chromebooks stay closed while your table talks. Sections 1 and 2 of the discussion form are already done. After you talk, finish Section 3 right here under My View. Submit before the exit ticket.
This form has three sections. Section 1: answer today's Board Question. Section 2: after the reading. Section 3 opens later in class. Today's Board Question: An AI helper does a student's homework for them. Name one person it helps and one person it could hurt.
CR 1. Board Question: An AI helper does a student's homework for them. Name one person it helps and one person it could hurt. Give your answer and one reason.
Write your answer in your lab notebook, labeled CR 1.
MC 1. Think about your answer to the board question. An AI helper does a student's homework for them. Name one person it helps and one person it could hurt. Which position is closest to yours now?
- It helps the student and hurts no one. The homework gets done.
- It helps the student today and could hurt that same student on the test.
- It hurts only the teacher, who has to grade work the student did not do.
MC 2. (Reading claim 1) A program can hurt people who never use it.
- Agree
- Disagree
MC 3. (Reading claim 2) An effect that the builders did not plan is an unintended consequence.
- Agree
- Disagree
MC 4. (Reading claim 3) An AI helper that gives answers always helps students learn.
- Agree
- Disagree
MC 5. (Reading claim 4) In the study, students with the standard helper scored higher on the practice problems than students with no helper.
- Agree
- Disagree
MC 6. (Reading claim 5) In the study, students with the version that gave hints scored higher on the exam than students who never had a helper.
- Agree
- Disagree
MC 7. (Reading claim 6) Amazon's résumé tool learned from ten years of résumés, and most of them came from men.
- Agree
- Disagree
MC 8. (Reading claim 7) Reuters reported that Amazon's résumé tool turned applicants down by itself, with no recruiter looking at its rankings.
- Agree
- Disagree
MC 9. (Reading claim 8) When Code Coach was asked to write a program that prints a name ten times, both of its answers contained Java code.
- Agree
- Disagree
CR 2. (A different view) Write the view at your table that was most different from yours. If everyone agreed, write the strongest argument against your table's answer.
Write your answer in your lab notebook, labeled CR 2.
CR 3. Think about the Board Question again: An AI helper does a student's homework for them. Name one person it helps and one person it could hurt. Do you have the same opinion as when you first answered the board question? If it is the same, what from the reading or table talk reaffirmed your opinion? If it is different, what from the reading or table talk changed your opinion?
Write your answer in your lab notebook, labeled CR 3.