DO NOT FOLD, SPINDLE, or MUTILATE: Broadening the Discussion about Educational Data (A Response to Iorio and Adler)
by Philip J. Piety - April 19, 2013
This is a response to a recent commentary that critiqued the use of data in education was leading to an obsession with quantitative reasoning and undermining core educational values. The author agrees with many of the concerns and presents an alternative argument drawing off of a new book titled “Assessing the Educational Data Movement” that frames the use of data in sociotechnical terms. He issues a call for inclusive dialog where the important perspectives of practitioners can be considered systemically alongside other values for educational data.
Jeanne Marie Iorio & Susan Matoba Adlers (2013) commentary Take a Number, Stand in Line, Better Yet, Be a Number Get Tracked: The Assault of Longitudinal Data Systems on Teaching and Learning takes aim at a number of issues, including the growth in educational data in state databases and the unique numbers used to identify and connect student and teacher records, saying:
Statewide longitudinal databases are becoming sources for decision-making by policymakers, administrators, and teachers. These databases are tracking children and teachers, reducing the performance of children and the work of teachers to numbers. We call for an end to the obsession with the quantitative and hope for a rethinking of assessment and teaching practices that trust children and teachers as capable and critical to learning, teaching, and assessment.
The authors also question an organization called The Data Quality Campaign (DQC) linking the DQC to several concerns about how collecting and using data is changing education. I agree with Iorio and Adler that this is an important topic. In a new book titled Assessing the Educational Data Movement published by Teachers College Press (Piety, 2013) I explore this time of change in American education and look specifically at relationships between different organizations involved in promoting the use of educational data. I studied the DQC in some depth and highlighted its mission that, in its own words, supports state policymakers and other key leaders to promote the development and effective use of statewide longitudinal data systems. I featured the DQC for two reasons. First, it is historically significant that an organization focused on the quality of educational data would emerge during this time. The DQC began around 2005, which makes it a relative newcomer in a field where some communities have existed for over a century. Second, the kind of information that the DQC provides is unique and important for understanding how data is and can be used. Much of educational research has historically been regional. Much of it, rightly so, has focused on districts, schools, teaching, and students. This has left a gap in understanding what is occurring in and across the different states and the DQCs publications provide important insights into a growing national educational information infrastructure. Understanding the role of organizations playing a role in the formation of national policy should be fair game. Being fair, as scholars, about their work and intentions is equally so.
With over 100 partnersincluding the Council of Chief State School Officers, The American Association of School Administrators, Consortium for School Networking, and the American Federation of Teachersthe DQC is now a fixture in the conversation about data in American education. It has received financial support from large philanthropies as well as encouragement and endorsements from the current and past Administrations and members of congress. It is a non-commercial entity with little connection to vendors. The DQC frequently hosts forums that bring together different points of view around data questions. I have attended over a dozen such events on topics that include data privacy and governance, using data to understand college and career-ready educational processes, and teacher effectiveness. I have never heard discussions about reducing any part of education to numbers. Rather there has been a deepening appreciation for the enormous challenges associated with making education a more information-driven field. A common phrase the DQC uses in its advocacy as been data should be used as a flashlight not a hammer. Recently the director of the DQC, Aimee Guidera (2012), wrote a commentary for Education Week titled Moving Beyond the Single Data Point that argued against releasing teachers names with value-added scores as had been done in New York and Los Angeles.
To be clear, I believe that asking questions about educational data is critical. The decisions made today in terms of infrastructures impact our childrens future and deserve a healthy debate. While there may not be technical limitations to including broad kinds of evidence in these systems, as Iorio and Adler suggest, those areas that are developed first will be better positioned later. The use of value-added models for math and literacy is directly related to those subjects being tested in response to NCLB. The kind of data collected today implicates what kinds of analyses are possible tomorrow. National conversations about what kinds of data to include should include many voices, including those represented by Iorio and Adler. The DQC similarly represents a set of values that are no less important. Before scholars or advocates attempt to place their views on a moral high ground, they might also consider the validity of other perspectives. Some believe the collection of certain kinds of data distorts education and increases inequity. Others believe the lack of data obscures educational practice where inequity is embedded. Education is a vast, complex endeavor and both of these interpretations are supported by at least some evidence. How we frame these differences and treat other perspectives have implications for the kinds of relationships we build. If we want to make real progress in our field, it is important to engage in a thoughtful and inclusive debate. I respect, but do not always agree with, Iorio and Adlers point of view when they say:
Reduced to unique identifiers, children and teachers are positioned as people with no voice and no valid contribution to how education might be imposed upon each of them [T]he teacher is explained through her teacher education program, her professional development experiences, and the performance of her students on a quantitative test How can we reduce the complexity of the child and teacher to numbers? Why are we ignoring the richness of relationships between teacher and student to quantify performance of children and credentials of teachers? How can policy and administrative decisions be based on this small and limited quantitative facet of the people a school system serves? Have the ease of numbers and belief that quantitative work is not biased become an easy way to forgo thoughtful decision-making?
Strong feelings about educational data are understandable because they can be so consequential for students and teachers. With each advance of technology comes concern about the changes they bring. It is hard to find a technologyfrom the printing press to steam engines to televisionthat has not brought apprehensions about what is slipping away. Students of Plato may recall he posited in 370 BC that the technology of writing would ruin the youth, saying:
[F]or this discovery of yours will create forgetfulness in the learners' souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves. The specific which you have discovered is an aid not to memory, but to reminiscence, and you give your disciples not truth, but only the semblance of truth; they will be hearers of many things and will have learned nothing; they will appear to be omniscient and will generally know nothing; they will be tiresome company, having the show of wisdom without the reality (Hackforth, 1972).
While a different time and technology, the tone is similar to what we hear from some about data. Many appreciate the aesthetic of old cars, period movies, and the pastoral charm of simpler days. Few people reading this, however, would consider living without refrigeration, personal computers, or modern vehicles with Bluetooth capabilities. Progress and change happen. My book argues that the use of data is just one way in which technology is moving education forward, but that the ways it is improving education are not always clear. I argue that much is missingspecifically an understanding of educational practice in classrooms and schools that Iorio and Adler representthat can inform the currently oversimplified views of educational data.
In the modern era, the development of digital records has given rise to fears about people being reduced to numbers and overreach of the state. Each year, most adults have various forms of communication that include a social security number (SSN). When the Social Security Act was new, many people were similarly alarmed by the idea of reducing people to numbers. Now, the SSN is prosaic and these numbers are usually assigned shortly after birth. If today the US Government entertained doing away with these common identifiers, many people would be alarmed and wonder instead how matters of public safety and finance could be managed without good record keeping?
In the 1960s, before optical character reading and electronic commerce, many checks and bills began to be printed on forms that could be read by the computers of that era. Many came with a statement that read DO NOT FOLD, SPINDLE, OR MUTILATE. The machines that used to read those documents would jam when the paper wasnt just right. In those early days some protesters would deliberately fold, spindle, mutilate (and even staple) those documents to upset the machine of the modern state. As successive waves of newer technology rendered those concerns moot, the phrase about folding, spindling, and mutilating became part of our cultural history (Lubar, 1992). We can resurrect it for at least two messages today. First, some of the immediate fears about new technology fade quickly with time and technological advances. While concerns about student and teacher identifiers may also recede, some of the related issues such as school accountability and evaluating teachers should not. Second, in advocating for our own point of view might we unwittingly fold, spindle, and mutilate the message and well intentioned work of others? I confronted this question in my book when discussing a number of organizations and projects that have attracted some polarizing views. Using principles of qualitative inquiry, I consulted with the organizations I wrote about, including the DQC, to give them an opportunity to clarify my impressions. Some may not agree with every part of this book. Hopefully, it will encourage fair and thoughtful discussion.
The world of educational data presents challenges for study. As Coburn and Turner (2011) said In many ways, the practice of data use is out ahead of research. Policy and interventions to promote data use far outstrip research studying the process, context, and consequences of these efforts. Educational researchers have some catching up to do. Researchers from other fields may find educational data a challenging area for them as well because it is not only a social practice, but social across many levels. These realities show up in the data; in its quality and consistency and in how difficult it is to use for comparisons (Piety, 2013). There are indeed many good and important questions about what kinds of artifacts are best to collect and how to integrate them into practices. As my book details, teachers and other practitioners have been far less represented in the design of educational data infrastructures than communities representing institutional interests. There is an opportunity for Iorio and Adler and those who agree with them.
One thing that Plato missed almost 2400 years ago and others are missing today is how new technologies work on levels beyond individuals. Much of the emphasis around data in education has been on how to improve individual students and teachers. The organizational and professional changes are less considered. Historically, sociotechnical revolutions from the printing press to television to the Internet have changed professions and organizational structures. Time and again, these kinds of revolutions can occur on many levels simultaneously, including individuals, teams, markets, and systems (Brynjolfsson, 1993; Cibbora, 1993). The history of writing (Harris, 1986) shows it had great impact on administrative structures and commerce. Blended learning school designsnew ways of structuring schoolsare also emerging during this data movement.
While it is popular to refer to other countries (e.g., Finland, Singapore, etc.) in terms of educational models, our situation with data is very American. Constitutionally, education is a state matter and local control yields a highly decentralized management of information. At the upcoming American Educational Research Association (AERA) meeting in San Francisco I have organized a symposium titled Big Data American Style. Our discussant is the incoming president of AERA and the authors of papers include faculty from Stanford University and Teachers College, a Vice President for Pearsons Center for Digital Data, Analytics & Adaptive Learning, the first Chief Privacy Officer at the United States Department of Education, and the Aimee Guidera of the DQC whose paper is titled Turf, Trust, Technical Issues, and Time: The 4Ts of Education Data Use and How Focusing on Meeting Peoples Information Needs Can Overcome Them. For many in educational research, the DQC will bring a new and important perspective. Will it be completely representative of all of the interests in educational data? Probably not. None will. We come together in the American spirit of encouraging diverse points of view to help collectively inform our understanding. Practitioner oriented researchers are welcome to come and ask thoughtful questions and help campaign for high quality discussions about educational data.
As a result of the exchange initiated by the March 8 commentary, Iorio and Adler and this author have begun a dialog about some important issues related to educational data, including equity issues and the uneven representation of different parts of the curriculum in the data movement.
Brynjolfsson, E. (1993). The productivity paradox of information technology. Communications of the ACM, 36(12) 66-77 .
Ciborra, C. (1993). Teams, markets and systems: Business innovation and information technology. Cambridge: Cambridge University Press.
Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9(4), 173-206.
Guidera, A (2012). What's in a Name? More Than a Single Data Point Commentary in Ed Week August 6. 2012. http://www.edweek.org/ew/articles/2012/08/08/37guidera.h31.html.
Hackforth, R. (Ed.). (1972). Plato: Phaedrus. Cambridge: Cambridge University Press.
Harris, R. (1986). The Origin of Writing. London, England: Duckworth.
Iorio and Adler (2013). Take a Number, Stand in Line, Better Yet, Be a Number Get Tracked: The Assault of Longitudinal Data Systems on Teaching and Learning. Teachers College Record March 8, 2013.
Lubar, S. (1992). Do Not Fold, Spindle or Mutilate: A Cultural History of the Punch Card. Journal of American Culture, 15(4), 43-55.
Mandinach, E. B., & Gummer, E. S. (2013). A Systemic View of Implementing Data Literacy in Educator Preparation. Educational Researcher, 42(1), 30-37.
Piety, P. (2013). Assessing the Educational Data Movement. New York, NY: Teachers College Press.