Regular readers know we take many so-called research reports on education with a grain of salt (See here and here for some examples). In general, the quality of education research has been shown to be low in a number of reports such as the National Reading Panel’s report of 2000.
Now, another really detailed report looking at well over 100,000 research papers in dozens of education journals has found that there is another reason to question much in education research.
“In a meta-study released in 2014 by the Educational Researcher, Matthew C. Makel of Duke University and Jonathan A. Plucker of the University of Connecticut, conducted a wide-ranging analysis of educational research to determine how many of the education studies published in the one hundred most prominent education journals had been replicated.
Of the 164,589 studies published in these education journals, only 221 of them were replications, an overall replication rate of .13%. Of the studies that were replicated, only 67.4% were successful, but 48.2%—nearly half of these replications—were conducted by the same people who did the original study, a bad research practice in and of itself.
In other words, less than 1% of education studies in the one hundred most prominent education journals meet the bedrock standard of replication, meaning that there is over a 99.9% chance that when someone quotes a study in the field of education, it cannot be relied upon as definitive.”
The ability to replicate the findings in a report is the bedrock of scientific analysis. It appears that rather than being built on bedrock, the education research community has built on sand, very unstable sand at that.
By the way, the Memoria Press’ take on the report isn’t unique. The Washington Post wrote of very similar impressions, as well.
I guess you could say the Memoria Press’ interpretation of the original report from Makel and Plucker has been replicated. That’s about the only replication that seems to be happening with education research.
By the way, this relates to our current concerns about a pending revision to Kentucky’s social studies standards. Defenders of that very deficient update will be quick to tell you that research shows what they did in the standards works. Well, take that with a grain of salt, too.