You won’t fix Kentucky’s reading problems by looking at the wrong ‘stuff’

Two months ago, the Bluegrass Institute released my report about While Kentucky’s Education System Was Sleeping…, which is loaded with comparisons of Kentucky’s performance to other states using the National Assessment of Educational Progress (NAEP) data. Beginning on Page 1 of this report, I explain that you cannot get a fair idea about how Kentucky really ranks for education compared to other states if you only look at overall average scores for all students. The reason is that student demographics now vary widely across states, so when you only look at overall average scores, what you really wind up doing is comparing a lot of white students’ results from Kentucky to a lot of results for students of color in other states. Thanks to the serious achievement gaps between whites and other student ethnic groups, the result for Kentucky will be inflated.

But some people never get the word.

Case in point came up during a recent meeting of the Kentucky Legislature’s Interim Joint Committee on Education. Leaders from the Kentucky Collaborative Center for Literacy Development (CCLD) made a presentation about methods of reading instruction that included an obvious attempt to defend the performance of a remedial reading program called Reading Recovery. While some of the data

presented will require more consideration, one set of data the CCLD provided involved the NAEP, and the CCLD got that presentation wrong.

Here is Slide 15 from the CCLD presentation, which is for NAEP Grade 4 Reading. It only considers overall “all student” average scores (Slide 16, which has more on NAEP, made the same mistake).

Figure 1. Slide 15 from the CCLD Presented to the Interim Joint Committee on Education on June 7, 2022

If you look at the CCLD slide, it shows Kentucky consistently outperforming both the national average and the Tennessee results for NAEP Grade 4 Reading since 2002.

Well, let’s look at the “Rest of the Story,” as the late Paul Harvey used to so nicely put it.

Figure 2 shows how the data trends when we take a more “Apples to Apples” look at white student scores in each state.

Figure 2.

Uh, Oh! This graph, assembled from data from the NAEP Data Explorer web tool, conveys a very different picture from the one CCLD presented.

Kentucky’s white students have never exceeded the US public school average on NAEP Grade 4 Reading. Never. And, far from never scoring lower than Tennessee, most recently (and also in the early years of KERA), Tennessee scored higher. However, this sort of simplistic comparison doesn’t tell you anything about the statistical sampling error in NAEP scores that can turn what looks like a difference into nothing more than a tie. If the CCLD said anything about that today, I missed it.

In any event, I checked the 2019 data, and that difference between Kentucky white student scores and the US public school average is statistically significant. By the way, in 2019 Kentucky’s public schools were 75% white while across the US public schools overall were only 46% white.

Now check out Figure 3, which covers Black student scores, which also was developed from data in the NAEP Data Explorer.

Figure 3

Believe it or not, Kentucky’s Black students did score ahead of the pack way back in the early days of KERA, but those days are now gone. In 2019, the most recent year of data available, Black students in Tennessee and US public schools overall both exceeded the Kentucky Black student score.

To be fair, when we consider the sampling errors in the Black students’ scores, in 2019 the scores are all statistical ties with each other.

But, back in 2015, Kentucky’s Blacks statistically significantly outscored both the US public school average and Tennessee for Black student scores. That was also the case in 2011. Kentucky outscored Tennessee by statistically significant amounts in several earlier years, as well.

So, the bottom line is Kentucky’s Blacks have lost ground in the most recent NAEP on Grade 4 Reading. The CCLD never mentioned that unfortunate truth. This brings me to a bigger puzzle. Why compare Kentucky to Tennessee when the state everyone is talking about where NAEP Grade 4 Reading is concerned is Mississippi?

Figures 4 and 5 will give you a clue as to why the CCLD might not want to look at Mississippi.

Figure 4

Figure 5

Note in Figures 4 and 5 that Mississippi started a real reading reform in 2013 and by 2019 was outscoring Kentucky for both white and Black student results – and both differences for 2019 are statistically significant. I’m fairly certain Mississippi no longer uses the Reading Recovery program.

At present, I’d rather follow what Mississippi is doing. They have real results that show they are on to something.

One more point: All the action from Kentucky’s Read to Achieve program, which uses Reading Recovery extensively, is finished by the end of Grade 3. So, the NAEP Grade 4 reading results come after Kentucky has realized all possible benefits from Read to Achieve and Reading Recovery. Once you realize that Kentucky’s NAEP Grade 4 Reading proficiency rates in 2019 were only 39% for whites and a really dismal 14% for Blacks, and those rates have been falling in recent years, it becomes obvious that despite anything else, the current early reading program in Kentucky, including all of its features — CCLD activities included — isn’t getting the job done and we need to look somewhere else.

Figure 6

For more insight on what is happening with reading in Kentucky, check out our other recent reports:

What Milton Wright knew about reading instruction, but lots of teachers apparently don’t 

and 

Reading proficiency rates rising in some Appalachian schools.

The second report shows Kentucky schools can do reading instruction a whole lot better if they get the right stuff to their teachers. To my knowledge, the CCLD

played no part in the surprising improvements covered in this second report. The CCLD presentation certainly never mentioned the remarkable progress in some of Kentucky’s Appalachian schools that we discuss in Reading proficiency rates rising in some Appalachian schools.

https://bipps.org/blog/you-wont-fix-kentuckys-reading-problems-by-looking-at-the-wrong-stuff

https://bit.ly/3uc4Ejn