PISA vs. PISA: What Works May Hurt
by Yong Zhao, Alma Harris & Michelle Jones — August 29, 2018
This commentary examines the side effects of PISA evidence-based policy recommendations.
Evidence-based has become the litmus test for educational policy and practice. It is predicated on an assumption that educational decisions based on scientifically collected evidence can bring the kind of improvement that has transformed medicine and other fields such as information technology and agriculture (Davies, 1999; Slavin, 2002, 2008). The past two decades witnessed a dramatic growth in efforts to use and establish evidence in educational policy and practice to identify and apply what works, as proven by evidence.
SIDE EFFECTS IN EDUCATION
In his latest book What Works May Hurt: Side Effects in Education (Zhao, 2018), Yong Zhao points out that an education product, be it a policy, program, or instructional method, can result in both positive and negative outcomes at the same time. Like medical products, they can cause damage while providing a cure. A medicine that is effective in treating heart problems for some people can be deadly to others. In other words, what works may hurt.
Modern medical research and practice has established a tradition of paying acute attention to both intended and positive effects as well as the unintended and adverse side effects of medical products. Unfortunately, side effects in education have been largely ignored. Efforts to collect evidence of what works are rarely accompanied with attention to the adverse side effects of what works. As a result, the evidence-based education policies and programs various governments have tried to implement (e.g., the Reading First program in No Child Left Behind) have not only failed to achieve expected outcomes but have also caused massive damage to those they were intended to help.
PISA VS. PISA
PISA has become the most powerful supranational enterprise that promotes evidence-based policies to educational systems. Relying on results from its tests and background surveys administered to 15-year-old students in an increasing number of countries, PISA has been prescribing evidence-based cures to all the worlds education ills for nearly two decades. In World Class: How to Build a 21st Century School System, Andreas Schleicher summarizes PISAs evidence-based recommendations for improving education. Below are some examples taken from the book.
Believing that all students [can] achieve at high levels (Schleicher, 2018, p. 71) is one of Schleichers recommendations. It is either meaningless because it does not suggest any specific policies or practices, or it is damaging if translated into specific policies and practices as advised by Schleicher, who essentially reduces the recommendation to clearly articulating the expectation that all students should be taught and held to the same standards (Schleicher, 2018, p. 71). Schleicher, based on his reading of PISA evidence, believes that external exams should be linked to national qualifications and drive curriculum and instructional practices. Writes Schleicher, Education standards and examinations are where the system of instruction begins, not where it ends. The key is how those standards and examinations translate into the curriculum, instructional material and ultimately instructional practice (Schleicher, 2018, p. 75).
Schleicher based this recommendation upon two major pieces of evidence. First, [t]he strength of the relationship between social background and the quality of learning outcomes varies substantially across education systemsproof that poor results are not inevitable for disadvantaged students (Schleicher, 2018, p. 39). He further cites evidence from Shanghai, Estonia, and Vietnam to support his argument that all students [can] achieve at high levels (Schleicher, 2018, p. 71). The second piece of evidence that Schleicher uses to directly back up his recommendation for standardized exams is that students in school systems that require standards-based external examinations score more than 16 points higher, on average, than those in school systems that do not use such examinations (Schleicher, 2018, p. 71).
First, believing all students [can] achieve at high levels (Schleicher, 2018, p. 71) can be a convenient excuse to ignore social injustice and individual differences, resulting in blaming the victims for not working hard enough. The concept that East Asian students have better PISA performance due to their belief in effort instead of talent, Schleicher cites in his book, has already been exposed as a myth that has perpetuated injustice for socially disadvantaged students and students with special needs (Zhang & Zhao, 2014; Zhao, 2014).
Second, the high-stakes external exams and centralized standards Schleicher recommended, while effective in driving up test scores in a limited number of subjects, have been found to carry serious adverse side effects including talent homogenization, suppression of creativity and innovation, anxiety, social-emotional well-being, and narrow educational experiences (Zhao, 2014, 2016, 2017, 2018).
Recruiting top graduates into teaching has been a widely promoted policy recommendation drawn from PISA data (Barber & Mourshed, 2007) because, as Schleicher reiterates, the quality of a school system will never exceed the quality of its teachers (Schleicher, 2018, p. 56). Schleichers presentation of this recommendation is first, confusing and second, self-contradictory. Schleicher brings evidence to show that actually students PISA performance can exceed their teachers proficiency: in some countries teachers proficiency in numeracy is average, but their students are top performers in the PISA mathematics test. In addition, in most high-performing countries, students score above what would be expected based solely on the average knowledge and skills of the teachers in those countries (Schleicher, 2018, p. 57).
Some other recommendations that Schleicher makes in the book are similarly problematic. For example, the suggestions that class size does not matter and spending more money does not lead to better outcomes could easily lead politicians to refuse to invest more in education and hire more teachers. Likewise, the suggestion to use examination systems as incentives for students to study harder can reduce and even extinguish students intrinsic motivation and love for learning.
WHY DO SIDE EFFECTS HAPPEN IN EDUCATION?
The problems with PISAs evidence-based recommendations arise from neglecting or simply ignoring the side effects. Side effects in education happen for a number of reasons. First, education has multiple goals, but the effects of an education intervention can be different on different goals. A policy or practice that is effective in accomplishing one goal can impede the realization of other goals. In the case of PISA, even if the factors that make an education system successful in achieving high test scores could be identified with precision, it is perfectly possible that they might hurt student well-being. Until 2015, PISA identified successful education systems based on student performance on its assessment. East Asian education systems, for example, have been found to be effective in producing excellent performances in tests and hurting students social emotional, psychological, and physical well-being, resulting in less life satisfaction, less positive attitude, and lower-levels of confidence.
Second, policies and practices that benefit some students can hurt others due to individual characteristics. Outcomes of treatments are always the results of interactions between characteristics of the treatment and of the individual or aptitude-treatment interaction (Cronbach, 1957; Cronbach & Snow, 1981). This is evidenced by the inconsistent finding about the effect of the quality of teachers in PISA because different qualities of teachers can have different effects on different students. The fact that PISA has found inconsistent impact of class sizes or long studying hours can be due to the possibility that small classes or long studying hours benefit some students but affect others negatively.
Third, policies and practices that work well in one context may cause harm in others. The unintended consequences of borrowing policies in education reform are well documented and evidenced (Harris & Jones, 2015). In the PISA case, this explains why some countries, actually only a very small number, have seen rapid improvements after adopting policies endorsed by PISA, while many others have not seen improvement. The countries performing less well after adopting PISA strategies tend not to be in the international limelight, and the side effects, or negative effects, of policy interference are less well known.
STUDYING SIDE EFFECTS
Clearly, just focusing on the evidence of positive effects of education products while neglecting their possible adverse effects can lead to harmful policies and practices, although they may be evidence-based. PISA has been a major provider of educational cures, but it has never discussed the side effects of its prescriptions. It is important that PISA, and other developers of evidence-based education solutions, begins to consider studying side effects in order to help bring safe and effective policies and practices. Until then, it is up to policy makers, education professionals, parents, and students to watch for negative side effects, because they exist.
Barber, M., & Mourshed, M. (2007, September). How the world's best-performing school systems come out on top. McKinsey & Company. Retrieved from: https://www.mckinsey.com/industries/social-sector/our-insights/how-the-worlds-best-performing-school-systems-come-out-on-top
Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12(11), 671.
Cronbach, L. J., & Snow, R. E. (1981). Aptitudes and instructional methods: A handbook for research on interactions. New York, NY: Irvington Publishers, Inc.
Davies, P. (1999). What is evidence‐based education? British Journal of Educational Studies, 47(2), 108–121.
Harris, A., & Jones, M. (2015). Transforming education systems: Comparative and critical perspectives on school leadership. Asia Pacific Journal of Education, 35(3), 311318.
Schleicher, A. (2018). World class: How to build a 21st-century school system. Paris, France: OECD Publishing.
Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 1521.
Slavin, R. E. (2008). Perspectives on evidence-based research in educationWhat works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 514.
Zhang, G., & Zhao, Y. (2014). Achievement gap in China. In J. V. Clark (Ed.), Closing the achievement gap from an international perspective: Transforming STEM for effective education (pp. 217-228). New York, NY: Springer.
Zhao, Y. (2014). Who's afraid of the big bad dragon: Why China has the best (and worst) education system in the world. San Francisco, CA: Jossey-Bass.
Zhao, Y. (2016). Who's afraid of PISA: The fallacy of international assessments of system performance. In A. Harris & M. S. Jones (Eds.), Leading futures (pp. 7-21). Thousand Oaks, CA: Sage.
Zhao, Y. (2017). What works can hurt: Side effects in education. Journal of Educational Change, 18(1), 119.
Zhao, Y. (2018). What works may hurt: Side effects in education. New York, NY: Teachers College Press.