Kentucky’s 2009 NAEP reading performance – is it credible?

I raised some questions yesterday about Kentucky’s fourth grade reading performance in the newly released 2009 National Assessment of Educational Progress (NAEP) Reading Report Card.

Today, I’m going to point out some more issues, this time focused on the state’s eighth grade results.

This graph shows results from the two most recent NAEP eighth grade reading assessments and the corresponding test results from Kentucky’s 100 percent testing of all eighth grade students with the EXPLORE Reading Assessment from the ACT, Incorporated.


Notice in the graph that Kentucky showed virtually no change in the percentage of students reaching or exceeding the EXPLORE Benchmark score between the 2006-07 and 2008-09 school terms.

In sharp contrast, Kentucky’s reading performance on the NAEP experienced a very notable five point rise in the same time interval.

Very simply, these notably different results raise questions about the 2009 NAEP scoring.

It should be pointed out that the EXPLORE test frameworks and scoring did not change, to my knowledge, during the time interval shown in the graph.

In sharp contrast, the NAEP reading assessment was changed in 2009 as explained by this quote from the 2009 NAEP Reading Report Card:

“The Reading Framework for the 2009 National Assessment of Educational Progress replaces the framework first used for the 1992 reading assessment and then for subsequent reading assessments through 2007. Compared to the previous framework, the 2009 reading framework includes more emphasis on literary and informational texts, a redefinition of reading cognitive processes, a new systematic assessment of vocabulary knowledge, and the addition of poetry to grade 4” (Page 4).

While the 2009 NAEP Reading Report Card goes on to claim that the 2009 results can be compared to prior year’s results, the Kentucky EXPLORE and NAEP comparison shown in the graph above raises questions about that assertion.

Clearly, the disparity between Kentucky’s NAEP and EXPLORE reading performance invites further investigation of the 2009 NAEP scoring. The situation also raises concerns about the validity of the 2009 rise in Kentucky’s NAEP scores in grade eight.

Furthermore, if the NAEP frameworks were changed in ways that impacted the trend lines for eighth grade, a similar problem could exist in grade four, as well. Unfortunately, ACT does not offer a companion EPAS test for that level.

The data shown in the graph for each test year is for the same cohort of students. Kentucky conducts EXPLORE testing in the fall of each academic year and the NAEP conducts its testing a few months later in late winter. Thus, it should be understood that the data in the graph above for 2006-07 and for 2008-09 are both for the same cohort of eighth grade students. NAEP didn’t conduct testing in the intervening school term.

It should also be emphasized that both the NAEP and the EXPLORE tests are controlled and scored by organizations outside of Kentucky. However, Kentucky does have some influence on the results from both tests because it has control on the use of testing accommodations. This results in some skewing of the results for both assessments.

For example, while few Kentucky students are excluded from EXPLORE, an appreciable proportion of the state’s students with learning disabilities receive test accommodations on EXPLORE that the ACT Incorporated does not allow on any ACT tests taken for college application purposes.

Kentucky’s accommodations program also impacts NAEP results, but in a somewhat different way. An appreciable proportion of Kentucky’s learning disabled students (48 percent in 2009 in grade four and 55 percent in grade 8) are completely excluded from the NAEP due to state requirements for incompatible testing accommodations that the NAEP will not allow. In fact, Kentucky’s learning disabled exclusion rate for NAEP reading in both fourth and eighth grade is among the very highest in the nation. Only two states had higher exclusion proportions in the 2009 eighth grade NAEP reading assessment.

This exclusion issue is largely due to a very controversial accommodation allowed in Kentucky – many of the state’s learning disabled students actually have all “reading” tests read to them. This totally changes the construct of the test from printed text decoding and comprehension to one of spoken word comprehension. Neither the ACT nor the NAEP will allow this clearly inappropriate accommodation on tests taken for record although EXPLORE does report scores for students tested this way for Kentucky’s internal state uses only.

Find more information:

About EXPLORE:

While many people are somewhat familiar with the NAEP, fewer probably know about the EXPLORE. Briefly, EXPLORE was developed by the ACT as a part of a coordinated college preparation testing program know as the EPAS system.

Kentucky tests all its students with EXPLORE in eight grade, the PLAN in tenth grade and then conducts 100 percent testing of all public school eleventh grade students with the ACT college entrance test.

Each of the EPAS tests has a set of “Benchmark” scores for the subjects of English, math, reading and science which are linked to an empirical study of how well ACT college entrance test scores compare to actual success in the first related college course. Students who score at or above the Benchmarks have a 75 percent chance of earning a “C” and a 50 percent chance they will get a “B” in those freshman college courses.

Learn more about the Benchmark Scores here.

ACT discusses the EPAS system here.

Information about the NAEP can be found in the 2009 Reading Report Card and in the NAEP Web Site.

Data sources used to assemble the graph:

2009 NAEP Reading Report Card, Table A-19

EXPLORE Excel Spreadsheet, “EXPLORE_Benchmark_06070910.xls” from Kentucky Department of Education.

Comments

  1. National math test scores continue to be disappointing. This poor trend persists in spite of new texts, standardized tests with attached implied threats, or laptops in the class. At some point, maybe we should admit that math, as it is taught currently and in the recent past, seems irrelevant to a large percentage of grade school kids.

    Why blame a sixth grade student or teacher trapped by meaningless lessons? Teachers are frustrated. Students check out.

    The missing element is reality. Instead of insisting that students learn another sixteen formulae, we need to involve them in tangible life projects. And the task must be interesting.

    A Trip To The Number Yard is a math book focusing on the building of a bungalow. Odd numbered chapters cover the phases of the project: lot layout, foundation, framing, all the way through until the trim out. The even numbered chapters introduce the math needed for the next stage of building and/or reviews the previous lessons.

    This type of project-oriented math engages kids. It is fun. They have a reason to learn the math they may have ignored in the standard lecture format of a class room.

    If we really want kids to learn math and to have the lessons be valuable, we need to change the mode of teaching. Our kids can master the math that most adults need. We can’t continue to have class rooms full of math drudges. Instead, we need to change our tactics and teach math via real life projects.

    Alan Cook
    info@thenumberyard.com
    http://www.thenumberyard.com

  2. niki hayes says:

    If you want to improve math scores, you have to improve the curriculum materials. When the radical shift to "meaningful" and project-based/group-oriented math was instituted in 1989 as "national standards" by the National Council of Teachers of Mathematics, with eyes focused only on girls and minorities, the whole discipline of mathematics was turned on its head. Calculators were put in the hands of kindergarteners and "boring drills" that provided immediate recall of multiplication facts, which is the basis of division and thus understanding fractions, were disregarded. These practice exercises weren't "fun."

    The "new new" math has been built on the idea that all children must discover math concepts and principles for themselves and teachers must act only as "guides." This is equivalent to having all music students learn to play an instrument by ear (discovery) rather than by learning the notes, their sounds, scales, etc.

    There is a place for learning the basic skills. That place is just not found in America's reformed fuzzy math books that have led 70% of our community college students and 40% of four-year university student into needing remedial math.

    By the way, I did use project-based activities in my middle and high school math classes as appropriate. It must be done judiciously and so that it provides real transcendence. Rarely does that happen in most classrooms. The activities are designed to entertain the students and make them "like" math. It's about the process, which becomes the "product." But the curriculum that my kids loved and which taught them real math was Saxon Mathematics, a "traditional" program.

Speak Your Mind

*