Maybe too good
The new National Assessment of Educational Progress results for the 2009 reading assessment were released today, and Kentucky’s overall performance really looks good.
We were the only state in the nation to post a statistically significant score increase in both fourth and eighth grade (NAEP does not report on state high school reading performance).
But, the results almost look TOO good.
For example, this map, found on page 13 in the new NAEP Reading Report Card, shows only one other state, Rhode Island, also made statistically significant progress in fourth grade reading since 2007.
But, hold on. All states have been pushing reading hard since No Child Left Behind was enacted in 2001. Can it really be that hardly any of them are making progress?
Furthermore, this table, which is extracted from a larger table found on page 14 in the Report Card, shows something that seems to conflict with what the map shows for Kentucky.
How is that possible?
Notice that in both Washington DC and Rhode Island, the only other places where reading progress supposedly was made, at least one racial group made progress. Not so in Kentucky. Strange, that.
The answer here may be due to some statistical sampling issues. Those statistical rules require larger score changes before statistical significance can be established when sampled groups are smaller.
However, with whites comprising 84 percent of the 2009 sample tested in Kentucky (Report Card, Page 52), it seems like if the overall scores made a statistically significant rise, then the whites should have a statistically detectable increase as well.
The fact that this didn’t happen makes me wonder if something unusual is happening in this NAEP scores report.
There is another curious thing. Overall, Kentucky’s average fourth grade reading proficiency rate on the NAEP was reported as 36 percent, while the national average proficiency rate was only 32 percent. However, when the disaggregated proficiency rates for whites and blacks are examined, things look quite different. Both Kentucky’s whites and blacks scored LOWER than their peers across the nation. Go figure!
For example, our exclusion rate for students with learning disabilities, once again, was well above the national average. That tends to inflate our scores because these particular students can be expected to score very low if they were to take the NAEP. Their exclusion abnormally raises our scores compared to other states.
In Kentucky, fully seven percent of all the students the NAEP wanted to test were ultimately excluded because they were determined to be too disabled to sit for a reading test. Across the nation, the exclusion rate for learning disabled students was only four percent, a decrease of one point from 2007. Kentucky’s exclusion rate was seven percent in 2007, as well.
Anyway, NAEP analysis has gotten quite involved, and simplistic examination of overall scores which some others like to engage in can be highly misleading. I’ll be looking at this some more, so stay tuned.