AI tutors are as good as human tutors -- but neither raises math scores

Human tutors are expensive. AI tutors are cheap. So it was big news when a British start-up claimed its AI-powered math tutor (with a human backup) could outperform a human tutor, writes Greg Toppo on The 74. But it's complicated. After 10 weeks of tutoring in 10 UK secondary schools, neither kind of tutoring improved scores on a standardized math test, a larger study showed. While girls showed some gains from tutoring, boys did worse.

"Eedi, along with Google DeepMind, found that their AI tutor matched expert human tutors at helping students fix mistakes in the moment," he writes. But the data don't show long-term gains.

Students worked on math problems one day a week in school, Toppo writes. "Whenever they got a question wrong, students were randomly assigned to one of four kinds of help: standard remediation in the app; a live, text-based chat with an expert human tutor; a chat with an AI tutor backed up by a human approving each interaction, or an AI tutor with “personalized context,” i.e., a summary of each student’s past misconceptions and work habits, such as whether they’d been rushing through problems."

Students who were tutored -- by AI or humans -- answered follow-up questions correctly 56 to 58 percent of the time, the study found. That compares to 43 percent after standard remediation.

But short-term gains didn't show up on a standardized math test: AI-tutored students did about the same as the control group, while with students with human tutors did slightly worse. That's probably because human tutors "covered less material in more depth," writes Toppo. Providing "personalized" information on students had no effect.

Girls who got AI or human tutoring did slightly better than girls in the control group, while "tutored boys actually did worse — especially those who worked with live human tutors," writes Toppo. Researchers speculated that talking with a tutor after every mistake may seem like a "signal of failure" to boys. And girls may do better with a text-based chat because they're more likely to be confident readers and writers.

Low achievers did better with the standard remedial practice than with human tutors.