by Nancy Perry & Kadriye ErcikanThe Programme for International Student Assessment (PISA) was designed by the Organization for Economic Cooperation and Development (OECD) to evaluate the quality, equity, and efficiency of school systems around the world. Specifically, the PISA has assessed 15-year-old students’ reading, mathematics, and science literacy on a 3-year cycle, since 2000. Also, the PISA collects information about how those outcomes are related to key demographic, social, economic, and educational variables. However, the preponderance of reports involving PISA data focus on achievement variables and cross-national comparisons of achievement variables. Challenges in evaluating achievement of students from different cultural and educational settings and data concerning students’ approaches to learning, motivation for learning, and opportunities for learning are rarely reported. A main goal of this themed issue of Teachers College Record (TCR) is to move the conversation about PISA data beyond achievement to also include factors that affect achievement (e.g., SES, home environment, strategy use). Also we asked authors to consider how international assessment data can be used for improving learning and education and what appropriate versus inappropriate inferences can be made from the data. In this introduction, we synthesize the six articles in this issue and themes that cut across them. Also we examine challenges associated with using data from international assessments, like the PISA, to inform education policy and practice within and across countries. We conclude with recommendations for collecting and using data from international assessments to inform research, policy, and teaching and learning.
by Cordula Artelt & Wolfgang SchneiderBecause metacognitive knowledge includes knowledge about adequate learning strategies, and an effective use of learning strategies is associated with higher levels of performance, substantial relationships can be assumed between metacognitive knowledge, strategic behavior, and performance. However, such a pattern of results is rarely found in the research literature. In part, this may be due to inadequate indicators of strategy use. In a representative sample of 15-year-old students in Germany, Artelt and Neuenhaus (2010) showed that high scores on self-reported strategy use were mirrored in high levels of performance only when students had sufficient metacognitive knowledge. Taking data from PISA 2009, which used a similar measure of metacognitive knowledge and also included self-report data on students’ strategy use, the present study aimed to test the cross-country generalizability of the relationship between metacognitive knowledge, strategy use, and reading competence for a total of 34 OECD countries. Results showed consistently moderate to high correlations between metacognitive knowledge and reading competence. There were also lower, but still significant, relationships between strategy use and both reading competence and metacognitive knowledge. Testing a “mediator model” with strategy use as a mediator resulted in small but significant effects of strategy use as mediator. Assuming that metacognitive knowledge might be a necessary precondition for effective strategy use, the study tested whether it served as a moderator. Results confirmed this moderator effect for many but not all countries. However, across all countries, there was a consistently high effect of metacognitive knowledge on reading competence, independent of the level of self-reported use of strategies. The discussion considers the validity of metacognition indicators (knowledge and strategy use) and practical implications of the findings.
by Hyo Jin Lim, Mimi Bong & Yeonkyung WooThe authors of this study examined how attitudes toward reading mediated the relationships between Korean adolescents’ reading environments and reading behaviors, using a nationally representative sample from the PISA 2009 database. Gender, home literacy resources, parents’ reading attitude, and parental support for reading were all significant predictors of Korean adolescents’ reading attitude. Having positive reading attitude, in turn, was positively associated with reading for pleasure, reading diverse types of materials, and application of various learning strategies.
by Ming Ming ChiuThis study examined whether microeconomic mechanisms can explain family and school inequalities and their relationships to students’ mathematics achievement. Multilevel analyses of 2009 PISA data from 475,760 fifteen-year-olds in 65 countries showed that students had lower mathematics achievement in countries with greater family inequality (GDP Gini) or school inequalities (of educational materials, teacher quality, or rich vs. poor schoolmates). Robustness testing of student subsamples (high vs. low socioeconomic status; high vs. low achievement) showed similar results; notably the richest students also had lower mathematics achievement in countries with greater family or school inequalities. The results suggest that these inequalities might operate through different mechanisms: (a) family inequality via fewer educational resources and via inefficient resource allocation, (b) school inequality of teacher quality via fewer educational resources, and (c) the school inequality of educational material results via inefficient resource allocation. These results suggest potential interventions to address each inequality mechanism.
by Kadriye Ercikan, Wolff-Michael Roth & Mustafa AsilTwo key uses of international assessments of achievement have been (a) comparing country performances for identifying the countries with the best education systems and (b) generating insights about effective policy and practice strategies that are associated with higher learning outcomes.
by Guillermo Solano-Flores & Chao WangWe investigated the relationship between illustration complexity and the difficulty of PISA 2009 science items for three participating jurisdictions. A negative correlation between the number of illustration features and item difficulty was observed for the United States and Mexico, whereas for Shanghai-China this correlation was negligible. Textual and object-and-background features shaped this correlation considerably. For the United States and Mexico, the magnitude of the negative correlation between illustration complexity and item difficulty was greater for larger numbers of different types of features. For Shanghai-China, this correlation was consistently close to zero. The pattern of magnitudes and directions of correlations observed parallels the three jurisdictions’ ranking in PISA 2009, which suggests that properly interpreting illustrations may play an important role in science achievement differences. However, given the exploratory nature of the study, this conclusion is tentative. We discuss the implications of our findings for systematically developing PISA science illustrated items.
by David C. BerlinerTrying to understand PISA is analogous to the parable of the blind men and the elephant. There are many facets of the PISA program, and thus many ways to both applaud and critique this ambitious international program of assessment that has gained enormous importance in the crafting of contemporary educational policy. One of the facets discussed in this paper is the issue of the comparability of the cognitions elicited by items across national and linguistic cultures. Valid interpretations of PISA results cannot proceed without assurance that items across nations are interpreted in the same way. A second facet examined is the association of PISA with economic outcomes for nations, still an unsettled area of importance. A third facet discussed is the search in PISA data for universally applicable instructional techniques, a possible will-o-the-wisp. A fourth facet examined are differences in cross-national attitudes toward the PISA subjects and how those affect test scores. Given these many facets of the program, a fifth facet that is arguably the most important of all the issues associated with PISA is discussed, namely the interpretation of PISA scores.
by Stephen SireciIn this article, I review and provide comments on the six articles that comprise this special issue on research conducted using PISA data. The articles represent a variety of issues and methods related to contemporary educational assessments and education policies. They feature state-of-the-art statistical analyses and instructive exploration of complex issues related to international assessment of students’ math, reading, and science achievement. A common theme underlying the articles is improving the interpretations of the results of educational assessments. Some articles address this theme via post hoc analysis or discussion of results, while others conduct research that informs future test development efforts.
by Maria Araceli Ruiz-Primo & Min LiA long-standing premise in test design is that contextualizing test items makes them concrete, less demanding, and more conducive to determining whether students can apply or transfer their knowledge. We assert, however, that despite decades of study and experience, much remains to be learned about how to construct effective and fair test items with contexts. Too little is known about how item contexts can be appropriately constructed and used, and even less about the relationship between context characteristics and student performance. The exploratory study presented in this paper seeks to contribute to knowledge about test design and construction by focusing on this gap. We address two key questions: (1) What are the characteristics of contexts used in the PISA science items? and (2) What are the relationships between different context characteristics and student performance? We propose a profiling approach to capture information about six context dimensions: type of context, context role, complexity, resources, level of abstraction, and connectivity. To test the approach empirically we sampled a total of 52 science items from PISA 2006 and 2009. We describe the context characteristics of the items at two levels (named layers): general (testlet context) and specific (item context). We provide empirical evidence about the relationships of these characteristics with student performance as measured by the international percentage of correct responses. We found that the dimension of context resources (e.g., pictures, drawings, photographs) for general contexts and level of abstractness for specific contexts are associated with student performance.