Does AI help students learn? We have no clue

AI helps students learn, according to 39 meta-analyses of research. Be dubious, reports

writes Hechinger's Jill Barshay.

“This is not trustworthy evidence,” said Justin Reich, director of MIT’s Teaching Systems Lab, at an Oct. 20 webinar. “All of these meta-analyses are flawed.”

A forensic audit of 14 of the 39 meta-analyses "found serious conceptual and statistical flaws," writes Barshay. One meta-analysis was retracted, almost a year after its May 2025 publication. 

A new audit by Patrick O’Neill, a professor at Ivy Tech community college in Bloomington, Indiana, found numerous errors, including “impossible statistics,” “hallucinated references” and “overstated conclusions,” he said at the seminar. "Some of the underlying studies even cited references to papers published about generative AI before that technology was released" and "links to the citations sometimes led to unrelated studies."

Irrelevant studies were included, writes Barshay. For example, "nursing students who studied for three hours a week with an instructor using AI were compared to peers who had no study sessions at all." The fact that students who studied did better says nothing about AI. "Students who played video games and answered AI-generated questions outperformed students who did neither."

O'Neill also warned of bias: Researchers asked for proof of AI's effectiveness, signaling their disinterest in studies that found it was ineffective or harmful.

O'Neill's own AI meta-analysis looked at 44 well-designed experiments, writes Barshay. The outcomes varied "from very helpful to very harmful to students."

Only 20 of 900 studies on AI in education were good enough to be included in a summary of the evidence for the AI Hub for Education, said Susanna Loeb, a Stanford education professor. Another 32 will soon be added.

"We did not identify any high-quality causal studies in K-12 settings in the U.S. for students and very few for teachers," the summary states. "Much of the research focuses on individuals above 18, is conducted in international settings, is studied in constrained conditions (such as a one-time 20-minute experiment) and focuses on short-term outcomes."

"AI tools significantly improve student performance on math practice, programming projects, and writing tasks," but benefits may be short term, the report states. AI tools can making learning easier -- "at the expense of deeper thinking." In addition, "tools designed with pedagogical guardrails (such as AI chatbots for tutoring that provide step-by-step reasoning instead of direct answers) show more promise than general-purpose AI tools."

It's too soon to do a useful meta-analysis, Loeb said.

Education research has become a "no-man's land," said Northwestern’s Beth Tipton. Peer review of new research relies on unpaid volunteers. There aren't enough experts looking at new claims. “Anybody can sort of say that they have evidence for anything, and share it with anybody, and consumers, schools, teachers, parents are left to try to wade through that and make sense of it. It’s incredibly frustrating.”

Be cautious about experimenting with AI, researchers advised educators.