OECD’s PISA: A Tale of Flaws and Hubris

In the latest math-focused PISA survey, Shanghai, Singapore, and Korea top the rankings, bumping three-time leader Finland to 12^th place, with Massachusetts and Connecticut (which were assessed separately for the first time) scoring significantly above the US average.

Responding to the US’s overall average performance, secretary of education Arne Duncan repeated his wake-up call of three years ago, calling on Americans to “face the brutal truth that we’re being out-educated.” PISA organizer Andreas Schleicher commented that the US education continues to tread water while Brazil, Germany, Poland, Singapore, and Shanghai have all pulled ahead.

The hand wringing about failing schools, sagging economic competitiveness, and the urgency of radical educational reform was repeated the world over. The Times Higher Education saw ‘East Asia on the Rise” and for the Guardian PISA proved that UK students still lag behind the rest of the world. In low-ranking Mexico the crisis mood was even more pronounced. Newspapers announced that PISA “shows a failed education system.” And Mexico’s president Nieto commented, in perfect OECD-speak, “that the conditions of education in Mexico are still far from what our students need and deserve to compete and triumph in a world that is increasingly more demanding and that requires higher competitiveness.”

THE DISNEYLAND-FACTOR

Apart from Finland’s sudden tumble to 12^th rank (difficult to explain given the country made no changes to its education practice or policy), the most eye-popping problem raised by the PISA rankings is that it derives all its rhetorical thunder from a profoundly skewed comparison. A majority of large, complex, ethnically or culturally heterogeneous education systems are held up to a deceptive comparison with a group of small city- and nation stations that are culturally homogeneous, and often politically authoritarian. Large and unwieldy systems like the United States (scoring 481), UK (494), France (495), Italy (485), Spain (484), are compared to spic-and-span city states like Singapore (scoring 573), whose government enjoys such arbitrary powers as the imposition of jail sentences on people who spit or chew gum in public.

The top group also includes Liechtenstein (PISA score 535), a “country” the size of Staten Island, sporting the highest per capita income in the world, with a total population of 35,000 and a high school population that would fit snugly into a single American urban high school.

Other top performers include Hong Kong (561), Macao (538) and, of course, Shanghai, accepted by OECD for the second time without question as the Chinese entry into PISA.

If we set these (quasi) city-states aside, the distance between the group of top performing countries and the group of alleged educational mediocrities like the US shrinks from a 120 point average to a 60 point average. If we further leave Korea and Japan aside to focus on Western democracies only, we find that high performing Canada (518), Germany (518), and Australia (504) outperform the putative educational laggards like the UK (494), France (495), or US (481) by some 20 or 30 points.

In other words, when comparing like with like, the differences shrink from a headline grabbing 120 points to a fairly unspectacular 20-30 points—roughly equivalent to the fluke point-loss sustained by three-time PISA darling Finland—a country that did not change its education practices or policies!

THE DOCILITY FACTOR

That the likes of Liechtenstein and Singapore are to the real world of public education what Disneyland is to a gritty urban playground is not the only problem confounding PISA results. Another problem is that top-performance in PISA also clusters around cultural and political idiosyncrasies which liberal democracies are unlikely to want to emulate.

Governments that get away with heavy fines for spitting and banning chewing gum (Singapore) or with jailing academics who call for more democracy (a word that drew a blank when I googled it in the library of Peking University a few years ago) naturally can wield educational clubs of a size that democratic nations lack. The effectiveness of these clubs is further magnified by long cultural habituation to cramming, all decisive make-or-break-entrance exams, and a system of shadow education that leave students little or no leisure time.

Thus we know that 88% of South Korean elementary school students and 61% of students in high schools receive private tutoring in cram schools. Private tutoring in South Korea represents 2.3% of GDP, equal to half the public education expenditures. In fact, while PISA holds Korea, Japan and Shanghai up to the rest of the world, many Koreans, Chinese, or Japanese take a much dimmer view of their schools, with their need for heavy out-of-school tutoring and the associated problems of depression, suicide, and a pervasive stifling of students’ academic self-motivation (Heyneman, 2013).

As someone who went to school in Germany when teachers there held unquestioned authority, I was amazed how the absence of that authoritarian attitude made the job of American teachers that much more difficult. On my visits to schools in China I have seen a mentality of collective docility much closer to my experience in Germany than in the US. With obedient students, teachers don’t need to spend time on discipline or “classroom management.” And while much of what we observe in the Eastern educational tradition is admittedly intriguing and enviable—deep respect for learning, reverence for the role of the teacher, and filial piety—these features often come on the back of less enviable characteristics like unquestioned obedience to authority, limits on free speech, and acceptance of paternalistic government we would be unwilling and unable to emulate.

This tradeoff between individual liberty and efficiency is routinely ignored, as the US and other of the alleged educational mediocrities are held up for comparisons to Shanghai, Singapore, or Korea. So let’s remind ourselves that as citizens of liberal democracies we give up a degree of orderliness and efficiency for the sake of the freedom of individuals to think for themselves, express their thoughts without fear, and self-organize in voluntary associations. That is part of the preparation for civic self-government that our schools are charged with and our courts regularly uphold. Despotic governments do not face these tradeoffs. They can literally “shanghai” their people into working overtime in preparation for a presumably all important test (Zhao & Meyer, 2013). And while the result may be higher test scores in 9^th grade math, there is no proof that those advantages translate into greater ability to come up with novel ideas and take risks in their pursuit—all features of an entrepreneurial attitude needed for economic prosperity. On the contrary, there is evidence that PISA success and entrepreneurial attitudes are inversely related (Zhao & Meyer, 2013). Nor is there any proof that whatever academic advantage the 15-year old inhabitants of authoritarian city states may enjoy carries over into higher education.

In fact, insiders often describe higher education in countries like Korea or Japan as marred by de-motivated students whose curiosity and ability for self-guided study has been numbed by years of high-stress drill. In a paper on Korea’s “examination wars” Korean sociologist Hae-Jong Cho reports that university courses “in liberal arts and social sciences that require analytical and critical thinking confuse and frustrate them endlessly. They are particularly annoyed by questions which do not have definite answers” (Heyneman, 2013, p. 297).

OECD AS GLOBAL EDUCATIONAL AUTHORITY

But the din about changes in rankings is masking the even more profound problem that in the wake of 15 years of PISA assessments, democratically elected national authorities are increasingly ceding control over their schools to an unelected, unaccountable body of OECD experts.

“PISA has become accepted as a reliable instrument for benchmarking student performance worldwide” is the conclusion of a recent OECD study on PISA’s global effect (Breakspear, 2010, p. 4). This goes beyond the well-publicized cases of “PISA shock” that led countries like Germany and Japan to better align their curricula with PISA requirements. The OECD study found that almost all 60+ governments used PISA to change their assessment and curriculum in order to “include PISA-like competencies.” As the US representative on the PISA-Board put it: “PISA has been assessed, along with other frameworks, in the formation of the new Common Core Standards” (p. 24), which now includes a strong emphasis on “reading competence” in decoding technical manuals and newspaper articles at the expense of understanding and interpreting works of literary merit.

There is little question that through PISA the OECD is reshaping the curriculum of public schools and the norms by which we judge them. The question is: should they?

The core of OECD’s mission has always been the growth of market economies, a mission that is safeguarded by member nations’ finance and economics ministers who govern the organization. In the course of the 1990s OECD has increasingly adopted a course of what critics call a neoliberal agenda of replacing public institutions with market mechanisms. Where international organizations like the United Nations Education and Scientific Organization (UNESCO) or TIMSS (Trends in International Math and Science Study) can claim a plausible mandate in matters of education, the OECD’s educational involvement is largely self-anointed, an appendix to its mission of capitalist market expansion.

Nor have we seen the end of it, as OECD is working on expanding its PISA franchise. In the next few years, the PISA brand will be complemented by several additional assessments like TALIS (Teaching and Learning International Survey), PIAAC (Program for International Assessment of Adult Competencies), and AHELO (Assessment of Higher Education Learning Outcomes). As the OECD seems hell-bent on assessing every square-inch of the educational globe, global education will soon resemble Bentham’s panopticon, in which education from pre-school to university is made transparent so that deviations from the global norm can be assessed—and sanctioned—with ease by a single human intelligence.

Looking at the overall effect of OECD’s self-selected expansion into public education, its main achievement may well be the transformation of the global education discourse. Framed by PISA, which OECD officials see as a “measurement of the flow of human capital into the economy” (Sellar & Lingard, 2013, p. 194), public education is less and less treated as a civic and cultural and increasingly seen as an economic project, a training ground for economic fitness.

OECD has become a global education authority with the ability to develop and impose “consensual knowledge about how the world works” and to change “the thinking of the people” (Wolfe, 2008, p. 41). With its increasingly exclusive monopoly on presumably objective educational data, OECD’s ability to “influence the running of societies is really, really good” as one OECD official put it (Sellar & Lingard, 2013, p. 196).

But while it is effectively influencing educational practice around the world, its policy and assessment regime is largely beyond the reach of democratic publics. Therefore, OECD’s PISA regime poses not only an educational, but also a political problem. The educational problem is one of global narrowing and homogenizing educational practices, and endangering valuable cultural diversity. The political problem involves a profound challenge to democracy due to a widening gap “between the totality of those affected by a political decision and those who participated in making it” (Held, 1995, p. ix).

To date, the education research community has taken PISA and OECD’s legitimacy largely at face value, duly dissecting the data it provided and debating policy options. It may be time to question OECD’s involvement in public education more fundamentally.

References

Breakspear, S. (2012). The policy impact of PISA. OECD Education Working Paper 71. Paris: OECD.

Held, D. (1995). Democracy and the global order. New York: Polity Press.

Heyneman, S. P.  (2013). The international efficiency of American education: The bad and the not-so-bad news. In Heinz-Dieter Meyer & Aaron Benavot (Eds.), PISA, power, and policy: The emergence of global educational governance (279-302). Oxford, UK: Symposium.

Sellar, S. & Lingard, B. (2013). PISA and the expanding role of the OECD in global educational governance. In Heinz-Dieter Meyer & Aaron Benavot (Eds.), PISA, power, and policy: The emergence of global educational governance (185-224). Oxford, UK: Symposium.

Wolfe, R. (2008), From reconstructing Europe to constructing globalization: The OECD in historical perspective. In Rianne Mahon and Stephen McBride (Eds.), The OECD and transnational governance (25-42). Vancouver, Canada: UBC Press.

Zhao, Y. & Meyer, H. D. (2013). High on PISA, low on entrepreneurship? What PISA does not measure. In Heinz-Dieter Meyer & Aaron Benavot. PISA, power, and policy: The emergence of global educational governance (267-278). Oxford, UK: Symposium.