Over the past two days I have been writing about the new results from the National Assessment of Educational Progress (NAEP). I have been stressing that the NAEP has a fair amount of statistical sampling error in all of its data, and that often turns what appear to be “wins” into nothing more than ties. Today, you will see how this important NAEP fact of life impacts what we can really learn from this assessment about the achievement gaps in Jefferson County Public Schools (JCPS).
We can look at JCPS NAEP data because this very large school district has participated in what the NAEP calls its Trial Urban District Assessment program since 2009. However, the sample sizes collected are fairly small, and that generates a considerable amount of statistical sampling error in the scores. That sampling error limits our ability to detect real changes in the district’s performance. Very simply, it takes more than a few points of difference in scores before we can validly conclude that a true change has occurred.
Sadly, an understanding of the statistical limits in the NAEP seem to have escaped some staffers at JCPS, because they made public claims about gap improvements based on the NAEP that are not really accurate. With one exception, what looks like “wins” in achievement gap improvements in Jefferson County are actually only ties with the gaps previously posted. As far as we can validly determine from the NAEP, Jefferson County cannot claim much progress with its achievement gap problems.