Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13

When Statistical Significance Hides More Than it Reveals

by Jeanne M. Powers & Gene V. Glass - July 02, 2014

Background & Purpose: The What Works Clearinghouse (WWC) recently released a summary of five CREDO studies of charter school outcomes produced between 2009 and 2013. We compared the WWC’s summary, which highlighted the statistical significance of the findings, to the effect sizes reported in the individual reports. We also addressed the findings reported in the original studies that were not summarized in the WWC reports.

Research Design: Analytic essay highlighting the gaps between the findings reported in the summary WWC report, the individual WWC reports for each study, and the original CREDO studies.

Findings: We argue that focusing on statistical significance is potentially misleading. The WCC summary invites the reader to conclude that charter schools had a greater effect on students’ achievement gains than traditional public schools. Comparing across the studies’ effect sizes suggests that the average effect of charter schools on students’ achievement gains is negligible. The WWC reports also do not address the considerable variation in achievement gains within and across subgroups of students and schools.

Conclusion: Summaries generated from research studies should provide an accounting of findings that allows practitioners to assess their practical importance. When these and similar reports are hard to understand and misleading, they run the risk of eroding practitioners’ trust in research and increasing rather than bridging the gulf between research and practice.

To view the full-text for this article you must be signed-in with the appropriate membership. Please review your options below:

Store a cookie on my computer that will allow me to skip this sign-in in the future.
Send me my password -- I can't remember it
Purchase this Article
Purchase When Statistical Significance Hides More Than it Reveals
Individual-Resource passes allow you to purchase access to resources one resource at a time. There are no recurring fees.
Become a Member
Online Access
With this membership you receive online access to all of TCRecord's content. The introductory rate of $25 is available for a limited time.
Print and Online Access
With this membership you receive the print journal and free online access to all of TCRecord's content.

Cite This Article as: Teachers College Record, Date Published: July 02, 2014
https://www.tcrecord.org ID Number: 17591, Date Accessed: 4/17/2021 4:18:53 AM

Purchase Reprint Rights for this article or review
Article Tools
Related Articles

Related Discussion
Post a Comment | Read All

About the Author
  • Jeanne M. Powers
    Mary Lou Fulton Teachers College at Arizona State University
    E-mail Author
    JEANNE M. POWERS is an Associate Professor in the Mary Lou Fulton Teachers College at Arizona State University. A sociologist, her research focuses on school segregation, school choice, and school finance litigation. Her publications include: “From segregation to school finance: The legal context for language rights in the United States” in Review of Research in Education (Volume 38) and Charter schools: From reform imagery to reform reality (Palgrave MacMillan).
  • Gene Glass
    Arizona State University
    E-mail Author
    GENE V GLASS is a Regents' Professor Emeritus from Arizona State University and a Research Professor at the University of Colorado Boulder. His most recent book is David C. Berliner, Gene V Glass & Associates, 50 myths and lies that threaten America's public schools, Teachers College Press, 2014.
Member Center
In Print
This Month's Issue