Assessing the Educational Data Movement

Tracing the recent acceleration of the educational data movement in the United States (US) from the enactment of The No Child Left Behind Act, Philip Piety conducts a sweeping examination of the movement, exposing both its shortcomings and its promise. In a highly readable book, he argues that while the educational data movement may be relatively young, fueled by policy initiatives, major foundations and technical developments, it is definitely here to stay.

Piety is less than impressed with many of the practices associated with the educational data movement and he provides a realistic and comprehensive appraisal of the terrain. He registers a wide variety of shortcomings which include: according to a report from a panel convened by the US Department of Education in 2009, available research on using data to make instructional decisions did not provide conclusive evidence of what works to improve student achievement—the generally accepted goal of using data; the use of test data within high-stakes accountability systems led to the adoption of undesirable practices, for instance focusing on “bubble kids” and narrowing the curriculum; millions of dollars have been spent on state longitudinal data systems (SLDS) without guidance from research or public input and, while SLDS have increased data capacity for states, sharing data across them is often problematic; and efforts to use data in teacher evaluation have been plagued with questions about reliability and fairness. For each shortcoming discussed across several chapters, Piety provides a thoughtful and wide-ranging analysis of the origins and consequences of these limitations. But Piety does not just present his readers with a host of problems. He also offers some new thinking and perspectives on directions for the educational data movement.

Among the approaches to solutions considered in the book is the comparison of business and education. Educators have long been exhorted to follow business practices with respect to data use—Walmart uses data to push for greater efficiency at all levels of its operations—education should follow suit.   But are the functions and products of business and education sufficiently similar to warrant these exhortations?  Using a model that incorporates three categories—executive management, peripheral components and technical core—Piety usefully draws parallels between the organizational structures of business and education but also points out where the two differ. Both settings are similar in that the executive management sets operational goals for the technical core and manages performance targets, and the peripheral components are those elements that any organization, whether business or education, needs to function effectively. The difference lies in the technical core of education, which is “vastly different from financial management or making products” (p. 25).  Despite these differences, Piety suggests that education could learn from business’ systems thinking, which routinely involves designing organizational information technologies that integrate the peripheral and technical core of the organization.  As a case in point, Piety cites Senator Michael Bennett of Colorado who worked in business before leading Denver Public Schools: “When I became superintendent our [human resources] HR department was essentially staffed by ex school administrators…busily not returning people’s phone calls and losing people’s paperwork and …[having] no idea how to do HR. That’s a systems problem that matters if you care about getting the most talented teachers into your school district. You can’t do this systems work unless you have data...and unless you are measuring what you are trying to do on all parts of the system” (p. 26).

To redress the absence of systems thinking in education, Piety proposes the use of organizational information technology (OIT), a technology that works across the organizational boundaries that are endemic to the technical core and peripheral components.  Drawing from research on business OIT innovations, he advocates for the use of OIT in education, not simply to help make yesterday’s activities more efficient, but rather to help reshape practices and structures towards greater effectiveness in the work of education as a whole.  Piety does not minimize the challenge of appropriating OIT into the practices of education, but his advocacy for its use is persuasive and will provide much food for thought for educators at all levels of the system.

Piety points out that a corollary of integrating components of the system will be the need for research that focuses on “the systematic aspects of education where interdepartmental function is realized.” He is critical of existing research on the use of educational data, noting that its fragmented nature means that it rarely leads to actionable conclusions, and sensibly calls for the research community to investigate the complex and interconnected realms of education practice.

This book is full of new ideas and fresh perspectives about educational data, but if it has a weakness, this lies in Piety’s treatment of data by teachers. Granted system thinking can bring about efficiencies of scale that ultimately impact the work of the classroom teacher, but may it not be said that it is the effective use of ongoing classroom data that shapes day-to-day teaching and learning in the classroom. Piety provides a cursory review of formative assessment, characterizing it as the use of short-cycle assessments. To be sure, he notes that these “assessments” may be embodied in “the things that the teacher already does: asks questions, gives quizzes and have students work on projects” (p. 97) and argues for the promise of digitally generated embedded assessments, stealth assessments and hidden assessments, to provide teachers with more data beyond basic math and literacy.  But despite citing Paul Black and Dylan William’s now famous review, he misses the larger point of formative assessment as a set of practices integrated into ongoing teaching and learning—clarifying learning goals, eliciting and interpreting evidence while learning is developing, providing feedback that moves learning forward, and involving students in the process through peer and self-assessment.  Effectively implemented, these are the practices that have the potential to increase personalized learning—something Piety views as desirable. Crucially, the implementation of formative assessment is dependent on initial and continuous teacher professional development, which many states in the nation are currently engaged in, but which Piety fails to mention.

After taking his reader through an extensive examination of the nexus of practices generated by the educational data movement—from data warehousing and analytics to models of teaching, to the teacher-student data link and value-added models, to digital tools in support of teaching and learning—Piety presents an optimistic picture of what the future might bring.  If the educational data movement is reoriented through the adoption of a marketplace framework, he suggests, its products will prove capable of satisfying a wide variety of users and purposes. A marketplace framework, he proposes, will enable educational data to be deployed interoperably across educational settings and to be exchanged and used by individual sectors of the system for different reasons. To make such an information exchange system possible requires an integrated design science in which the designs of educational approaches and the information systems that support them are considered together. Piety proposes five meta-principles for educational design science, the enactment of which will serve both to counter the current problems and fast forward the educational data movement into a productive future: 1) focus on systematic processes across boundaries; 2) recognize and design for data that is often opaque; 3) use robust information structures; 4) model temporality; and 5) design flexibly for social structures.  

Piety also has a final word for the research community. If researchers adopt a marketplace orientation they can investigate how the design of educational infrastructures can support not only “accountability, commercial interests, and practitioners, but research interests as well” (p. 184).

Piety breaks fresh ground with his insights about the possibilities of the educational data movement. Time will tell how ready and able the education community will be to embrace his vision of interoperability and multilevel data systems, and the question of data implementation in the engine room of the educational system—the classroom—remains to be more thoroughly explored.  Nonetheless, there is much to be learned from this important book.