I’m not the only one raising questions about uneven exclusion rates in the National Assessment of Educational Progress for students with learning disabilities.
Education Week’s Catherine Gewertz raises the issue in her March 25, 2010 blog item titled “Exclusion Rates in NAEP: A Pot Still Simmering.” You may have to scroll down a bit to find this as that blog gets more posts.
As Ms. Gewertz writes, the exclusion issue is indeed still simmering, and Kentucky has been at the epicenter of the discussion since 1999. At that time, I wrote a rather detailed paper about the situation. Using simple statistical analysis techniques, I found that the changes in exclusion that occurred in Kentucky and several other states could have resulted in score inflation that ran somewhere between slightly more than half a NAEP Scale Score point for each one percent increase in exclusion to as much as a full NAEP point for each one percent exclusion increase. In Kentucky’s case, where the exclusion rate reportedly rose from 4 percent of the raw sample to 10 percent between 1994 and 1998, there was evidence in the regression that all of the state’s six point score rise might have been due to nothing more than the exclusion rate change.
With NAEP reading score increases running all but flat, those exclusion-based score problems could be important.
Here are some links for this Thursday! Enjoy!
- New education portal on FreedomKentucky.org! We have revamped the hub of education information on our government transparency site. Click on the “Research” link and instantly find a wide variety of original research about state testing, teacher’s unions, and achievement gaps. Take a look!
- An interesting documentary that discusses citizens rights when encountering the police…
- The Kentucky Freedom Digest continues to keep an eye on policymakers, double-standards, and accountability in the Commonwealth. Check out their site!
- The Cato Institute discusses how attention will be turned to environmental issues in the aftermath of the Healthcare legislation.
I raised some questions yesterday about Kentucky’s fourth grade reading performance in the newly released 2009 National Assessment of Educational Progress (NAEP) Reading Report Card.
Today, I’m going to point out some more issues, this time focused on the state’s eighth grade results.
This graph shows results from the two most recent NAEP eighth grade reading assessments and the corresponding test results from Kentucky’s 100 percent testing of all eighth grade students with the EXPLORE Reading Assessment from the ACT, Incorporated.
In sharp contrast, Kentucky’s reading performance on the NAEP experienced a very notable five point rise in the same time interval.
Very simply, these notably different results raise questions about the 2009 NAEP scoring.
It should be pointed out that the EXPLORE test frameworks and scoring did not change, to my knowledge, during the time interval shown in the graph.
In sharp contrast, the NAEP reading assessment was changed in 2009 as explained by this quote from the 2009 NAEP Reading Report Card:
“The Reading Framework for the 2009 National Assessment of Educational Progress replaces the framework first used for the 1992 reading assessment and then for subsequent reading assessments through 2007. Compared to the previous framework, the 2009 reading framework includes more emphasis on literary and informational texts, a redefinition of reading cognitive processes, a new systematic assessment of vocabulary knowledge, and the addition of poetry to grade 4” (Page 4).
While the 2009 NAEP Reading Report Card goes on to claim that the 2009 results can be compared to prior year’s results, the Kentucky EXPLORE and NAEP comparison shown in the graph above raises questions about that assertion.
Clearly, the disparity between Kentucky’s NAEP and EXPLORE reading performance invites further investigation of the 2009 NAEP scoring. The situation also raises concerns about the validity of the 2009 rise in Kentucky’s NAEP scores in grade eight.
Furthermore, if the NAEP frameworks were changed in ways that impacted the trend lines for eighth grade, a similar problem could exist in grade four, as well. Unfortunately, ACT does not offer a companion EPAS test for that level.
The data shown in the graph for each test year is for the same cohort of students. Kentucky conducts EXPLORE testing in the fall of each academic year and the NAEP conducts its testing a few months later in late winter. Thus, it should be understood that the data in the graph above for 2006-07 and for 2008-09 are both for the same cohort of eighth grade students. NAEP didn’t conduct testing in the intervening school term.
It should also be emphasized that both the NAEP and the EXPLORE tests are controlled and scored by organizations outside of Kentucky. However, Kentucky does have some influence on the results from both tests because it has control on the use of testing accommodations. This results in some skewing of the results for both assessments.
For example, while few Kentucky students are excluded from EXPLORE, an appreciable proportion of the state’s students with learning disabilities receive test accommodations on EXPLORE that the ACT Incorporated does not allow on any ACT tests taken for college application purposes.
Kentucky’s accommodations program also impacts NAEP results, but in a somewhat different way. An appreciable proportion of Kentucky’s learning disabled students (48 percent in 2009 in grade four and 55 percent in grade 8) are completely excluded from the NAEP due to state requirements for incompatible testing accommodations that the NAEP will not allow. In fact, Kentucky’s learning disabled exclusion rate for NAEP reading in both fourth and eighth grade is among the very highest in the nation. Only two states had higher exclusion proportions in the 2009 eighth grade NAEP reading assessment.
This exclusion issue is largely due to a very controversial accommodation allowed in Kentucky – many of the state’s learning disabled students actually have all “reading” tests read to them. This totally changes the construct of the test from printed text decoding and comprehension to one of spoken word comprehension. Neither the ACT nor the NAEP will allow this clearly inappropriate accommodation on tests taken for record although EXPLORE does report scores for students tested this way for Kentucky’s internal state uses only.
Find more information:
While many people are somewhat familiar with the NAEP, fewer probably know about the EXPLORE. Briefly, EXPLORE was developed by the ACT as a part of a coordinated college preparation testing program know as the EPAS system.
Kentucky tests all its students with EXPLORE in eight grade, the PLAN in tenth grade and then conducts 100 percent testing of all public school eleventh grade students with the ACT college entrance test.
Each of the EPAS tests has a set of “Benchmark” scores for the subjects of English, math, reading and science which are linked to an empirical study of how well ACT college entrance test scores compare to actual success in the first related college course. Students who score at or above the Benchmarks have a 75 percent chance of earning a “C” and a 50 percent chance they will get a “B” in those freshman college courses.
Learn more about the Benchmark Scores here.
ACT discusses the EPAS system here.
Data sources used to assemble the graph:
2009 NAEP Reading Report Card, Table A-19
EXPLORE Excel Spreadsheet, “EXPLORE_Benchmark_06070910.xls” from Kentucky Department of Education.
Maybe too good
The new National Assessment of Educational Progress results for the 2009 reading assessment were released today, and Kentucky’s overall performance really looks good.
We were the only state in the nation to post a statistically significant score increase in both fourth and eighth grade (NAEP does not report on state high school reading performance).
But, the results almost look TOO good.
For example, this map, found on page 13 in the new NAEP Reading Report Card, shows only one other state, Rhode Island, also made statistically significant progress in fourth grade reading since 2007.
But, hold on. All states have been pushing reading hard since No Child Left Behind was enacted in 2001. Can it really be that hardly any of them are making progress?
Furthermore, this table, which is extracted from a larger table found on page 14 in the Report Card, shows something that seems to conflict with what the map shows for Kentucky.
How is that possible?
Notice that in both Washington DC and Rhode Island, the only other places where reading progress supposedly was made, at least one racial group made progress. Not so in Kentucky. Strange, that.
The answer here may be due to some statistical sampling issues. Those statistical rules require larger score changes before statistical significance can be established when sampled groups are smaller.
However, with whites comprising 84 percent of the 2009 sample tested in Kentucky (Report Card, Page 52), it seems like if the overall scores made a statistically significant rise, then the whites should have a statistically detectable increase as well.
The fact that this didn’t happen makes me wonder if something unusual is happening in this NAEP scores report.
There is another curious thing. Overall, Kentucky’s average fourth grade reading proficiency rate on the NAEP was reported as 36 percent, while the national average proficiency rate was only 32 percent. However, when the disaggregated proficiency rates for whites and blacks are examined, things look quite different. Both Kentucky’s whites and blacks scored LOWER than their peers across the nation. Go figure!
For example, our exclusion rate for students with learning disabilities, once again, was well above the national average. That tends to inflate our scores because these particular students can be expected to score very low if they were to take the NAEP. Their exclusion abnormally raises our scores compared to other states.
In Kentucky, fully seven percent of all the students the NAEP wanted to test were ultimately excluded because they were determined to be too disabled to sit for a reading test. Across the nation, the exclusion rate for learning disabled students was only four percent, a decrease of one point from 2007. Kentucky’s exclusion rate was seven percent in 2007, as well.
Anyway, NAEP analysis has gotten quite involved, and simplistic examination of overall scores which some others like to engage in can be highly misleading. I’ll be looking at this some more, so stay tuned.