I first wrote about problems with new additions to Kentucky’s Unbridled Learning school accountability program – known as the “Program Reviews” – some time ago. These problematic additions are self-scored by schools and thus are ripe for inflation in the same way Kentucky’s old Writing Portfolio element in the now defunct Commonwealth Accountability Testing System (CATS) and Kentucky Instructional Results Information System (KIRIS) assessments always provided questionable data.
Concerns about the Program Reviews surfaced again last week, becoming an interesting topic of discussion during the April 1, 2015 meeting of the Kentucky Board of Education.
Very simply, results from the state’s first three new education Program Reviews, which eventually will include five separate reviews for:
• The Arts and Humanities
• Practical Living/Vocational Studies
• Writing Instruction
• World Languages and
• The functioning of the Primary Program (Kindergarten to Third Grade)
don’t look trustworthy. Schools well-known for better performance sometimes got low Program Review scores while other schools with notoriously bad reputations had the audacity – or maybe just a lack of real understanding on the part of their less proficient staff – and self-award high scores for the quality of their education programs.
Especially for the Writing Program Review, where we can compare results to scores students receive for On-Demand Writing during KPREP testing, the Program Review scores for numerous Kentucky schools don’t look trustworthy (see more on that if you click the “Read more” link).
Program Reviews are a growing problem because starting with the 2013-14 school term the first three Program Reviews now count in Kentucky’s Unbridled Learning school assessment formulas. Inflation (in some cases – even deflation) in the Program Review scores not only provides bad information to policymakers and the public, but those errors now reduce the overall validity of the final scores from Unbridled Learning as well. The error could be considerable. At one point, the five proposed Program Reviews were going to count for as much as 30 percent of the overall Unbridled Learning score. The state board is now looking at reduced weighting, but providing any weight to these easily inflated items will certainly impact Unbridled Learning scores overall.
The Kentucky Board of Education is going to have to develop some better answers. Until that happens, it looks like scores from Unbridled Learning will probably suffer from more and more “Unbridled Inflation.” That isn’t going to help our students get better educations.
It also sets the stage for the same sort of problems that ultimately doomed the credibility of Kentucky’s former KIRIS and CATS assessments.