Does artificial intelligence help students learn? That’s a question many teachers and parents want answered, as the new educational technology is rapidly spreading in classrooms.
Normally, meta-analyses would be the right place to look, since they’re supposed to gather a scattered body of research and make sense of it by pointing to where the preponderance of evidence lies. And indeed, there are 39 of them, all reporting substantial benefits for AI in education.
But it appears that the studies synthesizing that evidence are themselves unreliable — and the peer-review process failed to catch the problems.
“This is not trustworthy evidence,” said Justin Reich, an associate professor at the Massachusetts Institute of Technology and the director of MIT’s Teaching Systems Lab, who hosted a webinar on Oct. 20 to discuss the problem of the growing number of positive AI meta-analyses. “All of these meta-analyses are flawed.”
A forensic audit of 14 of the 39 meta-analyses has found serious conceptual and statistical flaws. Appearing in journals by prominent publishers, including Springer Nature, Wiley, SAGE and Taylor & Francis, these 14 papers have accumulated more than 3,000 citations and continue to rack up 200 citations a month, according to Google Scholar — implying that AI’s effectiveness is settled science. One meta-analysis has been retracted, almost a year after its May 2025 publication in a Nature journal after two Norwegian scholars lodged a complaint.
The new audit was conducted by Patrick O’Neill, a community college professor at the Bloomington, Indiana, campus of Ivy Tech, who examined a sample of the underlying studies and the statistical methods used to combine their results in meta-analyses. O’Neill found recurring patterns of errors, some of which he said could easily have been detected just by reading the study abstracts.
During the Oct. 20 webinar, O’Neill described a constellation of “impossible statistics,” “hallucinated references” and “overstated conclusions.” Some of the underlying studies even cited references to papers published about generative AI before that technology was released to the public. Links to the citations sometimes led to unrelated studies.
Many of the aggregated studies didn’t even measure how much AI boosted learning. For example, nursing students who studied for three hours a week with an instructor using AI were compared to peers who had no study sessions at all (of course they tested better). Students who played video games and answered AI-generated questions outperformed students who did neither. In another experiment, students who chatted with an AI bot reported feeling less anxious.
Beth Tipton, a leading meta-analysis expert and a professor at Northwestern University, who joined her colleagues Reich and O’Neill in the critique, characterized as “crazy” the idea of conducting a very broad meta-analysis encompassing all the various uses of AI in education.
Tipton compared it to conducting a meta-analysis of math education. Is math education effective? That depends on how the math is taught. A thoughtful meta-analysis might narrow the question down to a specific approach, such as whether spatial learning interventions improve math skills. Similarly, Tipton said, how AI is used needs to be specified before conducting a meta-analysis of it.
“Do we mean people using [educational] apps and programs that include AI in them, like Khan Academy? Do we mean giving students access to Claude?” Tipton asked.
A big concern in the current batch of AI meta-analyses, O’Neill said, is “publication bias.” He found that two-thirds of the calls for papers for AI in education specifically invited research on its benefits, “seemingly disinviting null or negative results.” That means the meta-analyses are effectively “manufacturing evidence” by aggregating selected, positive results, he said.
Recrunching the numbers
O’Neill is now conducting his own AI meta-analysis, using proper statistical methods, and arriving at a far less sanguine picture. He found that the outcomes from 44 well-designed experiments varied widely, from very helpful to very harmful to students. The results averaged out to roughly zero, according to O’Neill’s preliminary calculations. This doesn’t mean educators should conclude that AI is harmless, on average. It depends on how it’s used.
Susanna Loeb, a Stanford professor of education who also joined the meta-analysis critique webinar, pointed out that her team has collected almost 900 studies on AI in education and determined that only 20 of them were of high enough quality to be included in a recent summary of the evidence for her AI Hub for Education. Her team has since identified 32 more studies and is soon to update that summary. But, she said, it’s still too early to conduct a meta-analysis that would be useful to school leaders or teachers. Very few of the high-quality studies take place in K-12 schools in the United States.
Northwestern’s Tipton worries that the fast-growing field of meta-analysis has outstripped the supply of experts qualified to review the work. More scholars are conducting meta-analyses because they’re quicker and cheaper than running experiments, and they can attract lots of citations for ambitious professors. The hunger for evidence about AI has made this topic a particular magnet for meta-analysis and spawned something of an AI meta-analysis cottage industry.
Tipton said she gets requests to review meta-analysis papers every day but can’t accept them all. Checking the underlying studies can take days, and peer review is unpaid volunteer work.
Medicine has flawed research, too, Tipton said, but it has gatekeepers to help separate the wheat from the chaff. The Food and Drug Administration evaluates evidence before approving drugs, and insurers review research when deciding which treatments to cover. Education has no comparable safeguards.
A ‘no-man’s land’
“In education, it’s kind of no-man’s land,” Tipton said. “Anybody can sort of say that they have evidence for anything, and share it with anybody, and consumers, schools, teachers, parents are left to try to wade through that and make sense of it. It’s incredibly frustrating.”
The researchers appealed to philanthropic funders and the National Science Foundation for help supporting rigorous, reliable research syntheses. “Maybe our model of voluntary peer review doesn’t work very well for landmark” studies on critical topics in education, Reich said. “Peer review’s not working. Meta-analysis is not working.”
Until truly reliable evidence emerges, the researchers advised educators to experiment cautiously with AI, starting with small groups of students and monitoring the results. Even introducing AI into a 300-student lecture course could be risky, O’Neill said.
In other words, teachers are on their own to experiment in their own classrooms. Still, it’s encouraging to see education researchers calling out flaws in their own field and trying to fix them.
Contact staff writer Jill Barshay at 212-678-3595, jillbarshay.35 on Signal, or barshay@hechingerreport.org.
This story about AI in education meta-analysis was produced by The Hechinger Report, a nonprofit, independent news organization that covers education. Sign up for Proof Points and other Hechinger newsletters.
More Proof Points
Republish our articles for free, online or in print, under a Creative Commons license.
