The Truth About Testing: An Educator's Call to Action

W. James Popham is a fixture on the circuit, appearing at every major conference and in almost every discussion about standardized tests.  His self-reference as a “recovering test developer,” is typical of the humor and easy style used in this readable synopsis of issues around state standardized tests.  In this book, he succinctly skewers current state test designs, calls for new content standards on which to base tests, and offers guidance for action to turn these recommendations into reality.

 

Standardized testing is the issue of the day.  Parents are excluding their kids from state tests.  Many teachers feel that they are sacrificing a broad curriculum to focus narrowly on test preparation.  Even more immediate is the implementation of the new federal “No Child Left Behind Act” which requires each state to expand its testing plan.

 

The Truth About Testing contains examples of poor practices that are all too prevalent.  It begins with an introduction outlining the history of standardized testing.  The litany of abuses is long: pressure on teachers to produce higher scores, resulting in “drill and kill” instruction (p. 21), cheating, and the misidentification of school status.  These evils are contrasted with the model of why we test, resulting in a picture of tests driving instruction, rather than assessments that measure important content that students have had the opportunity to learn.  The top two shortcomings of current tests are identified as teaching/testing mismatches (p. 43) and the tendency to jettison items covering important content (p. 46).  Popham also identifies “noninstructional factors” (p.73) that influence test scores including socio-economic status (SES) linked items.

 

The book is much more than a critique of state tests.  Popham outlines the elements of a model of how standards and assessments should be related.  He begins with the goal that the purpose of large-scale tests should be to provide “suitable instructional targets, as well as accountability evidence” (p. 76-77).  “Wish lists” of content standards should be pared to identify only the most important outcomes, because “it’s impossible to assess properly all the good things that we want kids to learn” (p. 78).  Sufficient items should be developed for each standard to “make a valid inference about a student’s mastery” (p. 82).  States should report on a standard-by-standard basis.  The items should require students to use the “complete array of enabling knowledge and subskills that task mastery requires” (p. 86).  An “assessment description for each content standard measured”, including illustrative sample items, is envisioned to document the linkage between content and test.  Evaluative criteria are provided with the idea that they should be used to review tests during development (p. 93-94).

 

There is a lively chapter on classroom assessment, including pointers on linking classroom tests to instructional decisions.  The chapter on collecting credible evidence of instructional effectiveness is a brief set of instructions on research design, offering advice about making the case about good things that are happening in the school.  The book ends with a call for action encouraging basic assessment literacy for policymakers.

 

I need to put my cards on the table.  I have spent several years in a state education department – in the state that implemented one of the first statewide tests.  I have coordinated development of a state curriculum framework that is the basis for our assessments.  I have most recently developed a state accreditation system using multiple measures to report school quality.  Our new state accreditation system is founded on the view that we cannot make judgments about schools based on a single test on a single day.   It appears that Popham and I agree completely about this premise.

 

Popham claims that state tests are often unfair because “there are meaningful mismatches between what is tested and what is supposed to be taught” (p. 77).  However, the whole point of having a set of standards is that there must be a defined set of things that all students need to know and be able to do.  This common set of content standards is the point to which districts align their locally developed curriculum.  Before state content standards, local curriculum was too often a guide that sat on a shelf.  Analyses such as the Third International Mathematics and Science Study (TIMSS) remind us that local curriculum decision-making often defaults to textbook selection.  Many teachers teach what they want to teach.  Assessments aligned to content standards are a reasonable way to promote a common core curriculum that all students should have the opportunity to learn.

 

Too many items on state tests are SES linked (p. 55).  However, there are many “break the mold” schools that far outperform expectations based on their demographics.  The effective schools literature charts this territory well.  The Education Trust has most recently lent its voice to the discussion, identifying schools that “dispel the myth.”  Popham’s discussion of SES linked items is a superficial overview that ignores this literature as well as test development strategies that address SES linkage through item review and piloting.  His argument leads many to outright dismissal of current tests.

 

The criticism of  “wish-list” content standards rings true, but the prescription of a narrowly defined set of assessable content could have unintended consequences.  Popham asserts that the content standards should identify  “the most important outcomes that an average teacher can teach with reasonable success” (p. 80).  He advocates further narrowing of this list to “only a teachable number of the field’s content standards (which) should be designated for high stakes assessment” (p. 80).  This recommendation is derived from the perceived atomic nature of curriculum, a belief that standards are taught and learned discretely, standard by standard.  However, many instructional units (and the individual lessons within them) address multiple standards.  One cannot simply sum the time required to teach each standard to determine what is teachable.  We do need focus and specificity in content standards.  Groups such as Achieve, Inc. should be applauded for their work in this direction.  The unintended effect of the contraction of assessed content will be a narrowing of the material that students are drilled on in test preparation efforts.

 

From the standpoint of curriculum and accountability policy, I applaud Popham for his candid observation that “the quest for wide score-spread tends to eliminate items covering important content that teachers have emphasized and students have mastered” (p. 48).   Some of the items based on “wish list” content standards are rejected because they are too hard and have low p values in pilot testing.  Popham correctly points out that items with high p values are also rejected, even though these items demonstrate student mastery of content that is important and valid.  Popham gives us grounds to rethink this practice.

 

Very few disagree that “content standards for instructionally illuminating tests should be few in number but as significant, clear, testable, and teachable, as possible” (p. 84).   We should take this as a call to put renewed focus on opportunity to learn in developing and evaluating content standards because the assessments can no longer be used to drive the curriculum.  We may never see tests that are truly worth teaching to.  However, tremendous advances have been made in assessment design to develop tasks and items that are much more authentic than we have ever had.

 

The new federal legislation asks states to expand their assessments to test all students in reading and mathematics.  A primary task in development of the new 3-8 assessments will be to focus specifically on important content at each grade level.  States should heed Popham’s call to “require test developers to think instructionally" (p. 80) so that the assessments are founded on important content reflecting authentic achievement.  The call for basic assessment literacy is an important first step toward this goal.