Mark ở vũ trụ khác · Another Universe Papers That Changed How We Learn #10 · AI in education · Maths

Does AI help or harm learning

Bastani et al. · "Generative AI without guardrails can harm learning: Evidence from high school mathematics" · PNAS 2025

PaperPNAS vol. 122, no. 26, article e2422633122 (2025)

Experimentnearly 1,000 high school students in Turkey, randomly split into 3 groups

Findingwith a chatbot practice improved; without it, exams were 17% worse

LabelCaution: well designed, open data, but new and not yet replicated

Source: Bastani et al. (2025), PNAS 122(26), e2422633122, abstract; the authors' public data (github.com/obastani/GenAICanHarmLearning). The label is Mark's assessment.

One-minute recap

Essence

Three groups, one schoolNo AI, a plain chatbot, and a chatbot with "guardrails" that only gives hints.

Practice: way upWith AI, practice scores rose 48% (plain chatbot) and 127% (hint chatbot).

Exam without AI: downThe plain-chatbot group scored 17% lower than students who never had it. AI became a "crutch".

Hints instead of answersA chatbot giving only teacher-written hints removed most of the harm, though exam scores did not rise either.

Design decidesAt Harvard, a carefully built AI tutor helped students learn more than an active-learning class.

Source: Bastani et al. (2025), PNAS abstract (48%, 127%, 17%, teacher-designed hints); Mark reran the main analysis on the public data and got the same numbers; Kestin et al. (2025), Scientific Reports 15, 17458, abstract.

Part 1

The paper's question

A chatbot can solve a homework problem in seconds. So if students use one during practice, do they learn more, or just finish faster?

Step 1Teacher lecturesAs usual, no AI.

→

Step 2PracticeThe control group works alone; the other two groups have a GPT-4 chatbot.

→

Step 3ExamAll three groups take a test without AI.

Source: Bastani et al. (2025), abstract; the practice (Part 2) and exam (Part 3) score structure follows the authors' public data and code.

Part 2

The story of the discovery

Who did it, where, why, and what was known before.

1984 · The dream of a personal tutorBloom wrote about the "2 sigma problem": one-to-one tutoring far outperforms a large class, but nobody can afford a tutor for every student. See Paper #8.

30/11/2022 · Chatbots for everyoneChatGPT launches. Students suddenly have a free "problem solver" available at any hour.

The Bastani team's questionHamsa Bastani (Wharton, University of Pennsylvania) and colleagues partnered with a high school in Turkey to test it in real classrooms, not a lab.

DesignGrades 9, 10, 11; 4 maths review sessions. Random assignment: no AI, "GPT Base" (like plain ChatGPT), "GPT Tutor" (prompts designed to protect learning, giving teacher hints instead of answers).

7/2024 · The preprint spreadsThe paper appeared on SSRN as "Generative AI Can Harm Learning" and was widely discussed before peer review.

6/2025 · Published in PNASThe new title adds two key words: "without guardrails". The team also released the data and code.

Source: Bloom (1984), Educational Researcher; ChatGPT launch date from Wikipedia (secondary source); Bastani et al. (2025), PNAS abstract; grades and sessions from the public data (final_data.csv); Wharton per Knowledge at Wharton (secondary source); SSRN 4895486 (7/2024); online date 25/6/2025 per the CHIBE page, University of Pennsylvania (secondary source).

Part 3

What the paper says, in 3 sentences

1. With AI, practice gets much betterPractice scores rose 48% with the plain chatbot and 127% with the hint chatbot.

2. Without AI, the plain-chatbot group fallsOn the AI-free exam they scored 17% below those who never had it: they used AI as a crutch.

3. Guardrails remove most of the harmWith a chatbot giving only teacher-written hints, exam scores were about the same as the control group.

Source: Bastani et al. (2025), PNAS abstract. Mark reran the main regression on the public data: +48.1%, +126.9%, −16.8%.

Practice and exam point in opposite directions

practice vs exam · raw meansfinal_data.csv

Doing the homework better is not the same as learning. The paper's official numbers (+48%, +127%, −17%) are adjusted for students' prior grades, teacher, session, grader and grade level. The raw numbers here only show the shape; Mark reran that adjustment and got the paper's numbers.Source: Bastani et al. (2025), abstract; final_data.csv and main_analysis.R in github.com/obastani/GenAICanHarmLearning (downloaded 15/10/2026), computed by Mark.

Why the plain chatbot became a crutch

crutch · answers vs hintsPNAS abstract; gpt_answers_full.csv

Two ways to ask, two results. The abstract names the key guardrail: asking the AI to give teacher-designed hints instead of answers. The 51% accuracy is Mark's count from the public GPT-4 answer file (57 problems, 10 attempts each; 237 logical errors, 43 arithmetic errors).Source: Bastani et al. (2025), abstract as listed on ADS; gpt_answers_full.csv in the authors' repository, counted by Mark.

Part 4

How this paper has been received

Each row has a source and lookup date.

Item	Finding
Venue, year	PNAS (peer reviewed), vol. 122, no. 26, article e2422633122, 2025. DOI 10.1073/pnas.2422633122. PNAS has published a correction (DOI 10.1073/pnas.2518204122); Mark could not read its content.
Citations	Not checked (Google Scholar blocked; no figure found in search results on 15/10/2026).
Replicated?	No replication with the same design found. Mark reran the analysis on the public data: matches 48%, 127%, −17%.
Comparison	Kestin et al. (Scientific Reports, 2025): a carefully built AI tutor helped Harvard physics students learn more in less time than an active-learning class.
Main criticisms	One school, one country, one subject, 4 sessions; the hint chatbot only removed harm, it did not raise exam scores; it used that year's GPT-4, newer models may differ (Mark's remark).
Awards	No source found, none listed.
Verdict	CAUTION good randomised design and open data, but new, one setting, not yet replicated.

Source: the PNAS page and PubMed 40560616 (via search results); github.com/obastani/GenAICanHarmLearning; Kestin et al., Scientific Reports 15, 17458 (2025). Criticisms are Mark's summary of the data and abstract.

Part 5

Fact check: facts and myths

14 things often said about AI and learning. Tap each row for the reason.

6 SOLID

6 CAUTION

2 FALSE

SOLIDPublished in PNAS 2025, vol. 122, no. 26

The authors' repository README, PubMed 40560616, IDEAS/RePEc. The full title adds "Evidence from high school mathematics".

SOLIDNearly 1,000 students, grades 9–11, 4 sessions, randomised

The abstract says "nearly a thousand"; the data hold 943 students, 4 sessions, grades 9, 10, 11. The main analysis drops honors classes, leaving 839.

SOLIDWith AI, practice scores rose 48% and 127%

PNAS abstract. Mark's rerun: +48.1% and +126.9%.

SOLIDWithout AI, the plain-chatbot group scored 17% lower

PNAS abstract. Mark's rerun: −16.8% adjusted. Raw: 29% vs 32%.

SOLIDGPT-4 at the time got almost half the problems wrong

Public answer file: 290 of 570 attempts correct (51%).

SOLIDKey guardrail: teacher hints instead of answers

A sentence in the paper's abstract (ADS version).

CAUTION"The hint chatbot made students do better on exams"

It only removed the harm: exam scores were about the same as the control group (Mark estimates −1.3%, effectively no difference).

CAUTION"Plain-chatbot students did not feel they learned less"

Reported in summaries of the study (secondary source); Mark could not read this part of the original.

CAUTION"An AI tutor at Harvard beat the classroom"

Kestin et al. (2025): true for 194 students in one intro physics course, two sessions, short term (design details per press, secondary source).

CAUTION"AI tutors double learning"

Kestin's 2024 preprint said "more than twice as much". The peer-reviewed version says only "significantly more in less time".

CAUTION"MIT proved ChatGPT rots your brain"

"Your Brain on ChatGPT" (2025) is an unreviewed preprint, 54 people writing essays with EEG; a published commentary points out inconsistencies.

CAUTION"In Nigeria, 6 weeks of English with a chatbot brought clear gains"

World Bank (De Simone et al., 2025): +0.23 standard deviations in English. A working paper, not peer reviewed.

FALSE"Using AI to study always does harm"

The hint-chatbot group was not harmed; Kestin's study found benefits from a carefully built AI tutor. Harm or help depends on how it is used.

FALSE"Meta-analyses have proved ChatGPT greatly improves learning"

The best-known meta-analysis (Wang and Fan, 2025, 51 studies, g = 0.867) was retracted by the journal on 22/4/2026 over discrepancies in the analysis.

Source: as noted in each row; full list under Sources.

Part 6

What the latest research says

5/2025 · Nigeria, 6 weeksWorld Bank: first-year senior secondary students studied English with Copilot (GPT-4) for 6 weeks in a randomised trial; +0.31 standard deviations on a composite assessment, +0.23 for English. Not peer reviewed.

6/2025 · HarvardKestin et al.: an AI tutor built on teaching principles; students learned more in less time than in an active-learning class.

2025 · "An effect in search of a cause"Weidlich et al. audited experiments in a large meta-analysis: only a small minority clearly described how AI was used, what the control group did, and measured actual learning.

22/4/2026 · A meta-analysis retractedWang and Fan's pooled analysis of 51 studies, once heavily cited, was retracted for "discrepancies in the meta-analysis".

Source: De Simone et al., World Bank Policy Research Working Paper WPS11125 (19/5/2025); Kestin et al., Scientific Reports 15, 17458 (3/6/2025); Weidlich, Gašević, Drachsler, Kirschner, Journal of Computer Assisted Learning 41(5), e70105 (2025); Retraction Note, Humanities and Social Sciences Communications 13, article 528 (2026). Read only as abstracts or notice pages via search results.

In short: the right question is no longer "is AI good or bad for learning" but "how is AI designed and used". Good evidence is still scarce.

Mark's reading of the studies above.

Part 7

Mark scores the paper

Scores are Mark's assessment based on the sources listed, not data from a study.

Experimental design

4/5

Generalisability

2/5

Practical meaning

5/5

Data transparency

5/5

Strengths

Randomised in real classrooms
Measured both with AI and after AI was removed
Public data and code, rerunnable

Weaknesses

One school, one subject, 4 sessions
No replication yet
A correction whose content is unclear

The paper does not say "AI is harmful". It says: AI doing the work harms, AI giving hints does not. A very clear lesson for learners.

Part 8

Lessons: use AI to ask for hints, not to copy answers

For students and self-learners. No AI expertise needed.

Try alone for 10 minutes firstOpen AI only when you have tried and know where you are stuck.

Ask for a hint, not the answerSay it plainly: "Don't solve it for me. Just give me a hint for the next step."

Doubt the AI's answerIn the study, GPT-4 got almost half the problems wrong. Check each step yourself.

Test yourself without AIAt the end, close the AI and redo 2 similar problems. That is your real score.

Source: "hints instead of answers" per the abstract of Bastani et al. (2025); 290/570 correct per the public data. The habits are Mark's suggestions.

Self-check after 1 week: 3 questions

1. How many times this week did I open AI before trying myself? Target: 0.

2. Did I ask for answers or for hints? Check my last 5 chats.

3. On an AI-free test at the end of the week, is my score close to my AI-assisted homework? If it is much lower, AI has become a crutch.

If AI does the work, the homework gets better. You do not.Mark's summary of the paper's result, not a quotation.

Want to learn to use AI for studying without it becoming a crutch? Visit Mark Học AI (markhocai.com) to read or learn.

Part 9

Sources

The working environment blocks PNAS, Nature, PubMed and Google Scholar. Titles, venues and abstracts were confirmed through those sites' own search results; the main numbers were checked by Mark against the authors' public data and code. Secondary sources are marked.

Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., Mariman, R. (2025). "Generative AI without guardrails can harm learning: Evidence from high school mathematics". PNAS 122(26), e2422633122. DOI 10.1073/pnas.2422633122. Correction: DOI 10.1073/pnas.2518204122. pnas.org · data and code
Kestin, G., Miller, K., Klales, A., Milbourne, T., Ponti, G. (2025). "AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting". Scientific Reports 15, 17458. DOI 10.1038/s41598-025-97652-6. nature.com
Wang, J. and Fan, W. (2025). "The effect of ChatGPT on students' learning performance, learning perception, and higher-order thinking: insights from a meta-analysis". Humanities and Social Sciences Communications. Retracted (Retraction Note, HSSC 13, article 528, 2026).
Weidlich, J., Gašević, D., Drachsler, H., Kirschner, P. (2025). "ChatGPT in Education: An Effect in Search of a Cause". Journal of Computer Assisted Learning 41(5), e70105.
De Simone, M. et al. (2025). "From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria". World Bank Policy Research Working Paper WPS11125.
Kosmyna, N. et al. (2025). "Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task". arXiv 2506.08872 (preprint).
Bloom, B. S. (1984). "The 2 Sigma Problem". Educational Researcher 13(6):4–16.
Secondary sources: Knowledge at Wharton and CHIBE Penn (authors, online date); Wikipedia (ChatGPT launch date); Hechinger Report and study summaries (Kestin design, students' perceptions); GovTech, Computing (retraction news).