Data-Driven Decision Making: Components of the Enculturation of Data Use in Education

THE CONTEXT

Much attention over the past decade has been given by policy makers to the importance of data-driven decision making and evidence-based practices in education (Aguerrebere, 2009; Cilbulka, 2013; CCSSO, 2012; Duncan, 2009, 2010, 2012; Easton, 2009, 2010; National Council for Accreditation of Teacher Education, 2010). And much has been written about data-driven decision making in recent years. The field has seen special issues of journals (Coburn & Turner, 2012; Turner & Coburn, 2012; Wayman, 2005a, 2005b, 2006), edited books (Herman & Haertel, 2005; Kowalski & Lasley, 2009; Mandinach & Honey, 2008; Moss, 2007), focused books (Boudett, City, & Murnane, 2007; Mandinach & Jackson, 2012; Supovitz, 2006), research syntheses (Hamilton, Halverson, Jackson, Mandinach, Supovitz, & Wayman, 2009), and federally funded landscape studies (Means, Chen, DaBarger, & Padilla, 2011; Means, Padilla, & Gallagher, 2010). Yet the field is still emerging. There is much we know (Hamilton et al., 2009), and there is also much we do not know (Mandinach, 2012; Marsh, 2012; Turner & Coburn, 2012) or that has methodological challenges. Even some of the most rigorously designed studies about impact result in interpretive questions (Carlson, Borman, & Robinson, 2011; Konstantanopoulus, Miller, & van der Ploeg, 2013).

As was aptly noted by Hamilton and colleagues (2009), there has been a great deal of research, much of it around case studies and implementation of data, but few studies that have reached the highest level of methodological rigor. In fact, this work reviewed nearly 3,000 documents and culled the research down to a precious few. This work is now over five years old. Additionally, Turner and Coburn (2012) and Marsh (2012) have raised criticisms of much of the research on methodological issues. Yet as Mandinach (2012) noted, the conduct of research in the area of data-driven decision making is difficult and challenged by the fact that research must mirror practice, and until there is sufficient practice to examine, the field will continue to struggle with the methodological and practical issues. The field still lacks definitive evidence that tests the logic model that data use positively impacts teachers’ practice and student performance (Carlson et al., 2011; Konstantanopoulus et al., 2013). However, that does not mean that ongoing research is not informative. Much can be learned from current research. Much of it is examining the foundational components of data-driven decision making.

Until recently, the field has lacked common terminology. Definitions of data literacy for policy (Data Quality Campaign, 2014) and research (Gummer & Mandinach, 2015, this issue) have emerged. The field has also has lacked psychometrically sound instrumentation to measure constructs such as use, beliefs, and data literacy (Mandinach, Trapani, & Gummer, in press; Wayman, Cho, Mandinach, Supovitz, & Wilkerson, in press). Work to address some of the foundational components of data-driven decision making will facilitate progress in research as well as practice and policy.

The IES Practice Guide’s (Hamilton et al., 2009) five recommendations also provide a foundation for the field: (a) create a cycle of inquiry; (b) make students their own data-driven decision makers; (c) develop an explicit vision for data use; (d) enculturate data use through the provision of necessary supports; and (e) provide a data system. These recommendations provide a concrete roadmap to the components needed for implementing data use in classrooms, schools, and districts. The components are the essential foundation for data-driven decision making. Even though the field recognizes the importance of leadership (Earl & Katz, 2006; Goldring & Berends, 2009; Halverson, Grigg, Pritchett, & Thomas, 2005; Knapp, Copland, & Swinnerton, 2007; Knapp, Swinnerton, Coplan, & Monpas-Huber, 2006), providing appropriate technology (Wayman, 2005a, 2007; Wayman & Stringfield, 2006), and providing ongoing professional development to build the human capacity to use data (Means et al, 2010), there is much that the field still needs to understand about these issues, particularly about the teachers who are invoking data-driven decision making in their classrooms.

Teachers are the heart of using data to inform instructional practice. Means and colleagues (2011) found that teachers have some understanding of how to use data but lack certain skills needed to be effective. They also found that teachers working in data teams help to compensate for individual teachers’ lack of competence with data. These findings help to ground the focus of many of the articles in this issue. A number of the articles deal with what data literacy is and how to build educators’ capacity to use data either in professional development or in teacher preparation. Other articles focus on the importance of teachers working in data teams or professional learning communities and social networking. Some of the articles review the existing literature, some are theoretical or conceptual, some are research based, and others focus on building human capacity through professional development. The intent here is to focus on teachers as a key component in the enculturation of data use, but the issue also recognizes the deeply systemic nature of the topic, including leadership, technology, and other supports and resources.

THE CONTENT

It is clear that data-driven decision making is a multifaceted enterprise. It is not a singular construct, but one with many interacting components. This special issue highlights research, theory, and practice about data use. The issue amasses some of the key names in data-driven decision making. The articles focus on some of the essential components and infrastructure needed to enculturate data into classrooms, schools, and districts. The articles can be categorized into those that focus on teachers, collaborative inquiry, and data teams. For teachers, the articles include data literacy, professional learning, and professional development. For collaborative inquiry and data teams, the articles focus on data coaches, professional learning communities, and networks.

Datnow and Hubbard provide a comprehensive review of literature on teachers' use of data. In spite of the massive push for data-driven decision making, the field still lacks a clear understanding on how teachers use data. Thus, how teachers actually use data to inform instructional practice and the factors that shape their decision making remain puzzling. This article reviews emerging research on what types of data are being used by teachers, how teachers make instructional decisions using data, and how their instruction and professional lives are impacted. Because district, school, and organizational contexts are also powerful shaping forces in teachers' use of data, this research review also discusses the factors that support or constrain teachers' use of data. Implications for further research and policy are discussed.

Mandinach, Friedman, and Gummer report on an empirical study that examines what schools of education are doing to build teachers’ capacity to use data. The study rests on the premise that educators must know how to use data and that schools of education are an essential venue for helping to build data literacy capacity. The study includes a survey of schools of education, an examination of syllabi, and a review of state licensure and certification requirements. The article examines the results and interprets them in light of a recognition that fostering change in institutions of higher education is difficult. The article discusses some of the challenges to integrating data use into the curricula, despite the acknowledgment and rhetoric from high-level education officials who are mandating the use of data, information, and evidence in educational practice. The article takes a systems approach to the complexities involved in implementing data use in higher education courses.

Bocala and Boudett describe the model of professional development used by the Data Wise Project at Harvard. They describe why data literacy for educators is important and how Data Wise addresses the need to develop data literacy, and they discuss the model’s signature pedagogy based on the work of Shulman (2005). The article outlines surface structure, deep structure, and implicit structure as they pertain to data literacy. The Data Wise pedagogical strategies include teamwork and receiving coaching, modeling and experiencing the data work, and application to one’s own context and reflecting on action. The article also addresses teaching data literacy at the “nexus of practice, policy, and research.” It concludes with a commentary on building capacity to develop data literacy at other institutions of higher education and training venues.

Gummer and Mandinach report on the work to develop a conceptual framework to undergird research, development, capacity building, and discussions around data literacy. Data literacy typically is interpreted as the collection, examination, analysis, and interpretation of data to inform some sort of decision in an educational setting. The field has been seeking both a definition of data literacy and a conceptual framework to inform research, capacity building, and policy making. For teachers and classroom data use, the capacity to use data is intimately linked to content knowledge and pedagogical content knowledge. Thus, this article presents a conceptualization of the construct, data literacy for teaching. The article is written with an eye to the ultimate objective of using the framework to ground development work for instruments to measure data literacy for teaching.

Jimerson and Wayman examine data-related professional learning for teachers. The article provides a conceptual framework that is grounded in prior theory and research. The article takes on an organizational perspective as it examines research on professional learning. It depicts both individual learning and collective learning. The article describes a study that examines requisite data-related knowledge and skills and how professional learning can be used to support those knowledge and skills. It considers school-level and district-level issues.

Gerzon scans the literature on data use, focusing on research that provides evidence related to the characteristics of effective school- and classroom-level data practices. Through this review, the article identifies a conceptual framework for school-level data use that includes the following five essential characteristics: participate in the flow of information for data use; provide resources for assistance with the social send-making process of data use; communicate professional expectation for data use; provide professional development on data use knowledge and skills; and provide leadership to nurture a culture of data use. Each characteristic is outlined to clarify key guiding principles from the research, examples of effective practices, and typical barriers. The purpose of this framework is twofold. The first is to integrate the elements that are, at this time, well documented and well understood in the research regarding effective school-level data use in order to provide guidance to school and district leaders. Second, this conceptual framework provides a new lens toward future dialogue and research related to critical data practices.

Marsh, Bertrand, and Huguet shed light on the supports that help literacy teachers use data to change instruction in classrooms serving Latina/o and African American students. They examine how coaches—data coaches and literacy coaches—and professional learning communities intervene in teachers’ data use in ways that sometimes lead to changes in instruction. In addition to learning lessons about what supports facilitated teachers using data to drive instructional change, the study also provides insights from the examples of data use that did not lead to change in instruction. The research also points to the importance of both horizontal and vertical expertise in helping teachers bridge the “knowing-doing” gap. However, the study also shows that several obstacles impede this process. In moving forward, the article calls for more research about how professional learning communities and coaches may help teachers use data to change instruction.

Farley-Ripple and Buttram explore the development of the capacity to use data from the perspective of social networks, focusing on the ways in which interactions between educators can support capacity building. The article presents a brief review of the data use literature, attending specifically to the issue of capacity. The authors identify two dominant views of capacity in the literature—capacity as situated at the individual and organizational levels—but argue that greater attention needs to be paid to a third conceptualization: capacity as embedded in social relations. They then examine this conceptualization of capacity in the context of an elementary school exemplified by its district as a strong user of data, focusing on the structure of data advice networks, the characteristics of key educators in the network, and the productiveness of the network in terms of influencing beliefs and practice. An important note is that this study is among the first social network analyses specific to data use and illustrates several ways in which this approach can be valuable in understanding individual and school capacity for data use.

Schildkamp and Poortman examine the impact of data teams from different secondary education schools in the Netherlands. The article captures some of the factors influencing data use in data teams. They authors also explore how school organizational, data, and participant characteristics influence the use of data in the data teams, particularly school leadership. The authors describe in depth how different factors promote or hinder the use of data in data teams. The study shows promising yet tentative results regarding the effectiveness of the data team approach in supporting schools in data use to improve education.

In the concluding article, Orland reviews the other articles in the issue and comments on them from the perspective of how the work on data-driven decision making can impact the field of educational practice and policy.

Acknolwedgments

The authors would like to acknowledge the contributions of Jeremy Friedman, who has served as a research associate on this work. The authors would also like to acknowledge Martin Orland for his contributions and support on this work. The authors would like to recognize WestEd’s administration for their support of the Data For Decisions Initiative. Finally, the authors would like to recognize other key individuals in the field of data use who have pushed our thinking, many of whom are represented in this special issue.

References

Aguerrebere, J. (2009). Panel discussion at Teachers’ use of data to impact teaching and learning. Washington, DC: Alliance for Excellent Education.

Bocala, C., & Boudett, K. P. (2015). Teaching educators habits of mind for using data wisely. Teachers College Record, 117(4).

Boudett, K. P., City, E. A., & Murnane, R. J. (Eds.). (2007). Data Wise: A step-by step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard Education Press.

Carlson D., Borman G. D., & Robinson M. (2011). A multi-state district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33, 378–398.

Cibulka, J. (2013, August). Leveraging the InTASC standards and progressions. Presentation made during the webinar Next Generation Standards & Accreditation Policies for Teacher Prep. Washington, DC: Alliance for Excellent Education.

CCSSO Task Force on Teacher Preparation and Entry into the Profession. (2012). Our responsibility, our promise: Transforming teacher preparation and entry into the profession. Washington, DC: Council of Chief State School Officers.

Coburn, C. E., & Turner, E. O. (2012). The practice of data use: An introduction. American Journal of Education, 118(2), 99–111.

Data Quality Campaign. (2014, February). Teacher data literacy: It’s about time. Washington, DC: Author.

Datnow, A., & Hubbard, L. (2015). Teachers’ use of assessment data to inform instruction: Lessons from the past and prospects for the future. Teachers College Record, 117(4).

Duncan, A. (2009, June 8). Secretary Arne Duncan addresses the Fourth Annual IES Research Conference. Speech made at the Fourth Annual IES Research Conference, Washington, DC. Retrieved from http://www.ed.gov/news/speeches/2009/06/06-82009.html

Duncan, A. (2010, November 16). Secretary Arne Duncan’s remarks to National Council for Accreditation of Teacher Education. Retrieved from http://www.ed.gov/news/speeches/secretary-arne-duncans-remarks-national-council-accreditation-teacher-education

Duncan, A. (2012, January 18). Leading education into the information age: Improving student outcomes with data. Roundtable discussion at the Data Quality Campaign National Data Summit, Washington, DC.

Earl, L. M., & Katz, S. K. (2006). Leading schools in a data-rich world. Thousand Oaks, CA: Corwin Press.

Easton, J. Q. (2009, July). Using data systems to drive school improvement. Keynote address at the STATS-DC 2009 Conference, Bethesda, MD.

Easton, J. Q. (2010, July 28). Helping states and districts swim in an ocean of new data. Keynote address at the Educate with Data: Stats-DC 2010, Bethesda, MD.

Farley-Ripple, E. N., & Buttram, J. (2015). The development of capacity for data use: The role of teacher networks in an elementary school. Teachers College Record, 117(4).

Gerzon, N. (2015). Structuring professional learning to develop a culture of data use: Aligning knowledge from the field and research findings. 117(4).

Goldring, E., & Berends, M. (2009). Leading with data: Pathways to improve your school. Thousand Oaks, CA: Corwin Press.

Gummer, E. S., & Mandinach, E. B. (2015). Building a conceptual framework for data literacy. Teachers College Record, 117(4).

Halverson, R., Grigg, J., Pritchett, R. B., & Thomas, C. (2005, July). The new instructional leadership: Creating data-driven instructional systems in schools. Paper presented at the annual meeting of the National Council of Professors of Education Administration, Washington, DC.

Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using student achievement data to support instructional decision making (NCEE 2009-4067). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides/

Herman, J. L., & Haertel, E. H. (Eds.) (2005). Uses and misuses of data for educational accountability and improvement (104th Yearbook of the NSSE, Part II). Malden, MA: Blackwell.

Jimerson, J. B., & Wayman, J. C. (2015). Professional Learning for Using Data: Examining Teacher Needs & Supports. Teachers College Record, 117(4).

Knapp, M. S., Copland, M. A., & Swinnerton, J. A. (2007). Understanding the promise and dynamics of data-informed leadership. In P. A. Moss (Ed.), Evidence and decision making (106th Yearbook of the NSSE, Part 1). Malden, MA: Blackwell.

Knapp, M. S., Swinnerton, J. A., Copland, M. A., & Monpas-Huber, J. (2006). Data-informed leadership in education. Seattle: University of Washington, Center for the Study of Teaching and Policy.

Konstantanopoulus, S., Miller, S. R., & van der Ploeg, A. (2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35(4), 481–499.

Kowalski, T. J., & Lasley, T. J., II. (Eds.). (2009). Handbook on data-based decision making in education. New York, NY: Routledge.

Mandinach, E. B. (2012). A perfect time for data use: Using data-driven decision making to inform practice. Educational Psychologist, 47(2), 71–85.

Mandinach, E. B., Friedman, J. M., & Gummer, E. S. (2015). How can schools of education help to build educators’ capacity to use data? A systemic view of the issue . Teachers College Record, 117(4).

Mandinach, E. B., Trapani, C. S., & Gummer, E. S. (in press). Measuring educators’ data literacy: The development of simulated assessments of data teams’ skills and knowledge. Washington, DC: WestEd.

Mandinach, E. B., & Honey, M. (Eds.). (2008). Data-driven school improvement: Linking data and learning. New York, NY: Teachers College Press.

Mandinach, E. B., & Jackson, S. (2012). Transforming teaching and learning through data-driven decision making. Thousand Oaks, CA: Corwin Press.

Marsh, J. A. (2012). Interventions promoting educators’ use of data: Research insights and gaps. Teachers College Record, 114(11), 1–48.

Marsh, J. A., Bertrand, M., & Huguet, A. (2015). Using data to alter instructional practice: The mediating role of coaches and professional learning communities. Teachers College Record, 117(4).

Means, B., Chen, E., DeBarger, A., & Padilla, C. (2011). Teachers’ ability to use data to inform instruction: Challenges and supports. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development.

Means, B., Padilla, C., & Gallagher, L. (2010). Use of education data at the local level: From accountability to instructional improvement. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development.

Moss, P. A. (Ed.). (2007). Evidence and decision making (106th Yearbook of the NSSE, Part I). Malden, MA: Blackwell.

National Council for Accreditation of Teachers Education. (2010). Transforming teacher education through clinical practice: A national strategy to prepare effective teachers. Washington, DC: Author.

Orland, M. (2015). Research and policy perspectives on data-based decision making in education. Teachers College Record, 117(4).

Schildkamp, K., & Poortman, C. (2015). Factors influencing the functioning of data teams. Teachers College Record, 117(4).

Shulman, L.S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 52–59.

Supovitz, J. A. (2006). The case for district-based reform: Leading, building, and sustaining school improvement. Cambridge, MA: Harvard Education Press.

Turner, E. O., & Coburn, C. E. (2012). Interventions to promote data use: An introduction. Teachers College Record, 114(11), 1–13.

Wayman, J. C. (2005a). Involving teachers in data-driven decision-making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Student Placed at Risk, 112(4), 521–548.

Wayman, J. C. (2005b). Journal of Education for Students Placed at Risk, 10(3). Special issue.

Wayman, J. C. (2006). Data use for school improvement [Special Issue]. American Journal of Education, 112(4).

Wayman, J. C. (2007). Student data systems for school improvement: The state of the field. TCEA Educational Technology Research Symposium (Vol. 1, pp. 156–162). Lancaster, PA: ProActive Publications.

Wayman, J. C., & Stringfield, S. (2006). Technology-supported involvement of entire faculties in examination of student data for instructional improvement. American Journal of Education, 112, 549–571.

Wayman, J. C., Cho, V., Mandinach, E. B., Supovitz, J. A., & Wilkerson, S. B. (in press). Development of a survey of data use. Alexandria, VA: CNA.