WWC Response to Powers and Glass
by Neil Seftor - October 29, 2015
Since 2002, the Department of Educationís What Works Clearinghouse (WWC) has conducted reviews of effectiveness studies in education. The WWCís reports are designed to help educators and policymakers make sense of the voluminous research in education and make more informed decisions about programs and practices. Among the more than 10,000 studies reviewed by the WWC, five from the Center for Research on Education Outcomes (CREDO) examine the effects of charter schools in different locations. This article seeks to address the concerns expressed by Powers & Glass (2014) regarding aspects of the WWCís presentation of the results of these five studies.
Since 2002, the Department of Educations What Works Clearinghouse (WWC) has conducted reviews of effectiveness studies in education. I am currently a director and co-principal investigator of Mathematica Policy Researchs WWC contract. As such, I have a thorough understanding of the WWC. The WWCs reports are designed to help educators and policymakers make sense of the voluminous research in education and make more informed decisions about programs and practices. Among the more than 10,000 studies reviewed by the WWC, five from the Center for Research on Education Outcomes (CREDO) examine the effects of charter schools in different geographical locations. This commentary seeks to address the concerns expressed by Powers and Glass (2014) in a research note regarding aspects of the WWCs presentation of the results of these five studies.
Each of the five CREDO charter school studies reviewed by the WWC appeared initially on their website as a Quick Review. The WWC conducts Quick Reviews of studies that receive significant media attention, and each of these five CREDO studies (released in 2009, 2010, 2011, 2012, and 2013 respectively) separately warranted such a review. It is unusual for the WWC to release numerous Quick Reviews on studies that are related. In this case, the same research team, using the same design and methodology, and examined overlapping samples created a single web page from which the five separate WWC reports can be accessed. This web page contains links to the WWC reports for all five CREDO studies, and briefly characterizes the findings from the studies. A summary table with a row for each study details the sample differences (years of analysis and grade levels), as well as direction of findings (positive or negative) for both math and reading.
The Powers and Glass research note summarizes their two main concerns with the WWC CREDO landing page. First, they argue that the summary table on the WWC website landing page is not an accurate representation of the magnitude of the study findings. They write:
[The table] invites the reader to count the number of cells with a + and come to the conclusion that, on balance, charter schools had a much greater effect on students achievement gains than traditional public schools [and] provides potentially misleading visual cues that may overemphasize and exaggerate the success of charter schools. (2014)
The WWC summary table provides the direction of impact on reading and math gains for the five studies, while the text notes that while there are impacts for all, only one study is statistically significant. This approach is consistent with the characterization of findings for other studies on the WWCs website landing pages. The WWC agrees that statistical significance alone does not provide information on the size of the impact, and provides the summary table as an overview of the more detailed findings (and magnitude of effects) that are reported in the individual reviews of each study. Though the website landing pages have been developed for a broad audience, the WWC is currently exploring the magnitude in a simple way, in response to the Powers and Glass research note (2014), WWC peer reviewers, as well as users of the website through the WWC Help Desk.
Second, Powers and Glass argue that the WWC reports do not address the complexity of the research findings. They write:
The WWC reports also do not address the considerable variation in achievement gains within and across subgroups and schools...these findings are substantially more complex and nuanced than the summary table might suggest, and are only minimally and inconsistently addressed in three of the five WWC reports and not addressed in two. (2014)
The website landing pages for all WWC reports provide a brief description of the overall findings, without describing supplemental findings. The rationale for this approach is to provide summary information for a general audience. The full reports present and discuss additional findings, such as those from analyses of subgroupsonly if those additional analyses also meet WWC standards. Thus, some Single Study Reviews of CREDO charter school studies contain additional information about subgroup outcomes and some do not, depending on whether or not the analyses met WWC standards.
As a director and co-principal investigator of a WWC contract, the WWC appreciates the concerns of Powers and Glass, with regard to the presentation of the five CREDO charter school studies. Presenting the CREDO charter school studies on the WWC website was an unusual challenge for the WWC, but one that was important to undertake for practitioners and other audiences. While having a single web page from which multiple WWC reports on the same topic can be accessed is unique, the presentation is consistent with other WWC approaches to providing information. In creating reports and brief summaries for the web, the WWC must balance the amount of information that is provided with the level of accompanying explanation. Therefore, the brief web summaries have focused on high-level characterizations of overall findings, while the accompanying reports provide more detail on findings and analysis.
The WWC has grown and evolved since 2002, and encourages questions and comments about our reviews and products through the WWC Help Desk, which can be reached <a href="http://ies.ed.gov/ncee/wwc/ContactUs.aspx">through this link</a>. Anyone can request information from the Help Desk about meeting WWC standards, asking questions about WWC procedures or processes, finding out how to become a WWC-certified reviewer, and alerting the WWC to possible errors in reports. When errors are identified, reports are corrected and reposted. The WWC continuously strives to improve its reviews and contribute towards important educational conversations. We appreciate the research note produced by Powers and Glass and invite other educational stakeholders to improve the effectiveness of education as a whole.
Powers, J. M., & Glass, G. V. (2014). When statistical significance hides more than it reveals, Teachers College Record. Retrieved from http://www.tcrecord.org/Content.asp?ContentID=17591