Is this related to testing problems?
A January 31, 2014 letter from Kentucky Governor Steve Beshear and the state’s education leaders again focuses national attention on the Bluegrass State’s implementation headaches with the Common Core State Standards. For sure, as the leading state for Common Core implementation, headaches here are better developed than anywhere else in the nation.
The Beshear letter says:
“Kentucky anticipates that it may have to engage in a Request for Proposal (RFP) process as part of the next wave of assessments for the Commonwealth. Kentucky state law requires a fair and equitable RFP process. We want to ensure that PARCC has the opportunity to participate in this process as a potential bidder, if it wishes to do so, without any perception of a conflict of interest or bias being created. Accordingly, it makes practical sense for Kentucky to withdraw from PARCC, before any competitive bid process is initiated.”
Potential bidding preferences aside, Kentucky’s new action in the testing scene raises a number of questions, none more serious than the indication that the state is dissatisfied with its current Common Core State Standards testing program and may be getting ready to rebid it.
The state’s current Common Core State Standards based testing program is called the Kentucky Performance Rating for Educational Progress (KPREP). It includes two separate elements, grade-by-grade testing in math and reading in grades three to eight, currently provided by Pearson Publishers, and high school End-of-Course (EOC) testing from the ACT, Inc.
I discussed the notable problems with the EOC program – total removal of open-response questions and collapse of online testing last year – in the first blog on this topic.
Now, I examine initial test results from the first two years of grade three to eight KPREP, which – while limited – does raise concerns about scoring inflation.
To begin, we have two years of testing data from KPREP for 2012 and 2013. That gives us both scores and the beginning of a trend line.
We can compare that KPREP information to Kentucky’s performance on the National Assessment of Educational Progress (NAEP) reading and math assessments for fourth and eighth grade students. Unfortunately, NAEP was not offered in 2012, but it was offered in 2011. So, I can “sandwich” the KPREP data within NAEP data for nearly the same time frame. Because Kentucky’s NAEP data has not been changing much in short intervals, this gives us a useful, though admittedly not perfect, comparison.
Now, examine this first graph, which shows how the NAEP and KPREP in Grade 4 reading compare over the past few years.
Notice that for each racial group, the first data bar shows the 2011 NAEP proficiency rate. Then, the two years of KPREP data for 2012 and 2013 are displayed. The “sandwich” for each racial group is completed with the 2013 NAEP testing results.
Let’s look at the most disturbing information in the graph first, which is the right hand set of bars for Kentucky’s Hispanic student performance. NAEP in 2011 reported that 35 percent of Kentucky’s fourth grade Hispanics read proficiently. Two years later, the NAEP reported a notable, 6-point drop in their reading proficiency rate to just 29 percent.
Now, look at what the two “sandwiched” KPREP scores show. Per KPREP, Hispanic reading improved from 36.1 percent to 37.2 percent proficiency between 2012 and 2013. The fairly close agreement between the NAEP in 2011 and the KPREP in 2012 was erased in 2013 when the proficiency gap between the two tests increased to more than eight points.
A similar situation occurred for Kentucky’s black students. NAEP showed a 4-point drop in reading proficiency. KPREP reported a slight improvement. The disparity between the 2011 NAEP and the first year of KPREP testing was 7.5 percentage points. It rose to 12.1 percentage points in 2013.
In contrast, the amount of change in Kentucky’s White trends in Grade 4 reading tracked pretty well between the two assessments, but the discrepancy in proficiency rates was already high. In 2013, the same cohort of whites only were 39 percent proficient on the NAEP, but KPREP claimed more than half, 52.3 percent were proficient, a gap of 13.3 percentage points.
Here is a similar graph covering the eighth grade math comparison. The differing trends and the growth in KPREP inflation are quite notable here for all three races.
Taking the data in order from left to right this time, Kentucky’s white proficiency rate on NAEP Grade 8 math stayed completely flat at 33 percent between 2011 and 2013. In notable contrast, KPREP math testing reported a nearly 4-point rise in proficiency for Kentucky’s whites over nearly the same time interval. In 2013 the NAEP-KPREP proficiency gap was 15 percentage points.
Blacks experienced a 1-point decline in NAEP proficiency over the same time frame while KPREP reported a rise in math performance for these students. As of 2013 the reported Grade 8 math proficiency rate discrepancy was 13.4 percentage points for Kentucky’s blacks.
Hispanic KPREP-NAEP discrepancies were even more pronounced. While NAEP shows a statistically insignificant 1-point drop in proficiency, KPREP claimed a substantial improvement of more than 5 percentage points. The gap in Hispanic proficiency rates reported by NAEP and KPREP in 2013 was 22.2 percentage points, a very serious difference pointing to considerable inflation in the KPREP scores.
I also have graphs for Grade 4 math and Grade 8 reading along with overall student averages, as well. Click the “Read more” link to see those. The situations are not quite as dramatic as those in the graphs above, but KPREP is still generally over-reporting proficiency compared to the NAEP.
And, that might also explain why Kentucky’s educators seem to be sending signals that they are planning assessment changes after only two years of KPREP operations. It also might explain why a bill to end Kentucky’s involvement, while of uncertain future, does have multiple sponsors in the legislature.