The Prichard Committee’s Blog weighed in on March 11, 2012 about Kentucky’s nation-leading levels of exclusion of students with learning disabilities in the National Assessment of Educational Progress (NAEP) reading assessments.
Sadly, Prichard’s blog glossed over the exclusion problem, claiming, “Kentucky’s recent record of relative NAEP success does not, in fact, evaporate when exclusion rates are considered.”
Well, that’s just not so. In Part 1 I discussed some very important testing policies, which Prichard ignored, that led to Kentucky’s NAEP exclusion problem. Now, I’ll discuss technical deficiencies in Prichard’s analysis of NAEP scores for those students who do not have learning disabilities.
To begin, I need to point out, again, that you have to look at NAEP data in the right way. You cannot get a fair picture of what is happening with state education programs using simplistic state to state comparisons only of “all student” NAEP test scores. The reason is student demographics now vary dramatically from state to state. In many other states high minority populations drag down the overall state average scores. Thanks to shifting racial demographics, it is actually possible for a state to make progress for each of its student racial groups but still fail to improve its overall average “all student” scores. There is even a name for this phenomenon: Simpson’s Paradox.
Because of this well-known problem, the most recent guidance from the NAEP 2011 Reading Report Card (See page 24) says:
“Differences in states’ demographic makeup should be taken into consideration when interpreting state results” (emphasis added).
Prichard’s simplistic analysis doesn’t do that.
Take a look at what happens as soon as we disaggregate the data by race to see how non-learning disabled white fourth grade students in Kentucky match up to the national average reading scores for their counterparts in other states.
That is exactly opposite of what the “all student” analysis used by Prichard shows.
Proving the wisdom in the NAEP guidance about doing state to state comparisons, Prichard’s assertions regarding Kentucky’s reading performance for students who are not learning disabled start to fall apart as soon as the disaggregated data is examined. It isn’t appropriate to only conduct a simplistic analysis and then claim Kentucky’s reading performance is better than the national averages. In fact, Kentucky’s reading performance for the state’s dominant population of students is worse than elsewhere.