A couple of days ago the Kentucky Department of Education sent out a glowing press release, “Report Ranks Kentucky in Top Ten for Core Academic Improvement.”
The release crowed about how well Kentucky looked in a new report from Harvard, “Achievement Growth: International and U.S. State Trends in Student Performance.”
I’m told the Maryland department was cheering about that state’s supposed performance, as well. Certainly, the Maryland story was picked up by the Washington Post.
In both cases, the claims were based on the Harvard team’s review of score improvements in reading, mathematics and science from the National Assessment of Educational Progress (NAEP).
Well, I know Kentucky’s NAEP performance isn’t quite so Sterling; so, I took a look at the Harvard report to see how the researchers developed their findings. What I found was no surprise.
As usual when we hear these sorts of amazing claims about Kentucky’s public education system, the Harvard report only examines overall average student scores from the NAEP. The report thus ignores cautions found in all recent NAEP Report Cards that when data is compared between states, you have to consider the racial makeup of the school systems in those states and make note of unusual rates of exclusion of students from taking the NAEP, too.
The Harvard report uses some rather mysterious calculations (more on that later, if you click the “Read More” link below) to show Kentucky’s overall academic growth on NAEP between 1992 and 2011 across the three subjects of reading, mathematics and science ranked at number 7 among 41 states with NAEP data (See Table B.2 in the report)(rank updated 28 Jul 12 to correct for an error in Kentucky Department of Education’s press release, which claimed Kentucky ranked 5th).
But, the report’s methodology, which, I reiterate, ignores warnings in the NAEP Report Cards themselves, overlooks some very important facts. And, the report creates an overly rosy picture for Kentucky, as well.
For example, as the table below shows, Kentucky’s black fourth grade students certainly didn’t share in all this wonderful success, at least not in math.
To read this table, notice that the 2011 scores appear first, followed by the 1992 scores and the difference in those scores (change). Where other states have the same change as Kentucky, the score change and the ranks are highlighted in yellow to signify a tie. (Note added 25 Jul 12 – This table ignores the statistical sampling error in the NAEP scores, which would further ‘fuzzy up’ the results, creating more ties for Kentucky but less differentiation between the states, as well.)
Among the 34 states and the District of Columbia that had reported grade four math scores for blacks in both 1992 and 2011, Kentucky wound up ranking way down near the bottom of the listing for progress in a four-way tie for 27th place. Only five states had a slower rate of progress for black students on NAEP Grade 8 Math than Kentucky did.
The situation is nearly the same for Kentucky’s eighth grade blacks in math, by the way. Kentucky ties for only 20th place out of 33 states with useable NAEP data for blacks.
The Harvard report’s authors claim that in most states that raised average scores, scores also went up for both top achieving students and low-achievers, too. In other words, as the report says on Page 23, “In most states, a rising tide lifted all boats.”
Well, Kentucky’s blacks clearly missed that boat. But, you have to do more than limited research to learn that. You have to look beyond the “all students” NAEP scores before you can see that.
And, even if Kentucky’s whites did post somewhat better improvements (though definitely not in even the top 10 for math, let alone the top 5), the grim reality, as the map below shows, is that as of the 2011 NAEP Grade 8 Mathematics Assessment, Kentucky’s whites still scored very close to the bottom of the heap even after making that improvement.