Teaching Educators Habits of Mind for Using Data Wisely

by Candice Bocala & Kathryn Parker Boudett - 2015

Background: Institutions of higher education, specifically schools of education, should play a pivotal role in supporting educators’ development of data literacy for teaching. While novice teachers are often prepared to use test-based assessment data, they learn these experiences in isolated courses that do not connect to instruction or school improvement. Moreover, once these novice teachers begin working in schools, they are increasingly expected to work with colleagues to apply data literacy skills, yet few preparation programs provide sustained support with using data collaboratively for whole-school improvement.

Purpose: This essay describes the habits of mind, or ways of thinking and being, that underlie data literacy courses offered by the Data Wise Project at the Harvard Graduate School of Education. The habits include: shared commitment to action, assessment, and adjustment; intentional collaboration; and relentless focus on evidence. Adding an emphasis on habits of mind expands building data literacy beyond accumulating discrete knowledge and skills or learning a process that becomes routine.

Research Design: The authors provide suggestions for instructional design than can be incorporated both in degree-program courses and in ongoing professional development. These suggestions provide opportunities for participants to actively cultivate the three habits of mind.

Conclusions: In order to support all educators while learning data literacy for teaching, there is a need to bridge the resources of an institution of higher education with the instructional capacity of professional development providers and the authentic experiences of school-based practitioners.

Educators¹ are increasingly responsible for using multiple sources of data in collaborative conversations about student learning and school improvement decisions (Coburn & Turner, 2011, 2012; Marsh, 2012). Learning to be an educator no longer just encompasses the acquisition of deep content knowledge and flexible application of pedagogy; it also means learning to work with colleagues to use evidence of student learning to make instructional decisions. The phenomenon of data inquiryreferred to variously as data-informed decision making and using data for improvementhas become pervasive in schools (Means, Chen, DeBarger, & Padilla, 2011; Young & Kim, 2010). We define data inquiry as educators working in teams to analyze student progress using data, make recommendations about curricular and instructional next steps, and follow up on the results of these actions (Hamilton et al., 2009; Kekahio & Baker, 2013; National Forum on Education Statistics, 2012). There is much attention paid to the need for educators to gain timely access to multiple forms of data and learn how to use them for decision-making and instructional conversations (Supovitz, Foley, & Mishook, 2012; Wayman, Jimerson, & Cho, 2012). If there is a growing consensus that strong educator practice now involves having professional conversations about student learning and instruction, then it would make sense for preservice preparation programs and ongoing professional development to build educators capacity to participate in such practices. Across the field of education, we do not have clarity about the role that institutions of higher educationespecially schools of education that train novice teachers and school leadersmight play, particularly in contrast to the role of professional developers who work with practicing educators (Mandinach & Gummer, 2013). However, it is becoming clearer that schools of education can play an important part to ensure that teachers enter into schools well prepared to engage in data inquiry (Mandinach, Friedman, & Gummer, 2015, this issue).

This article offers the authors perspective as teachers and learners at the Data Wise Project at the Harvard Graduate School of Education,² which has worked closely with educators for over a decade to develop teaching strategies to support data inquiry in schools. It begins with an overview of the need to build data literacy as part of preservice and ongoing professional learning for educators. It then describes the three habits of mind around data use that educators need the most support in developing. It describes how Data Wise courses are designed to give participants hands-on opportunities to cultivate these habits. The focus is on illuminating the authors rationale behind the instructional choices in hopes of sparking a conversation about teaching educators how to engage in data inquiry. The article concludes by pointing out the need to bridge the resources of an institution of higher education with the instructional capacity of professional development providers and the authentic experiences of school-based practitioners.

PREPARING EDUCATORS TO BE DATA LITERATE

Currently there is interest in the field around supporting all educatorsboth those in preservice training and those already working in schoolsto become more data literate, or able to understand, analyze, and act on multiple forms of data about student learning (Turner & Coburn, 2012). Learning to use data to inform instruction requires sophisticated content, pedagogical, and professional knowledge (Young & Kim, 2010). Collectively, these capacities have been called data literacy skills (Mandinach & Gummer, 2013). Many experts agree that data literacy includes problem-focused skillsknowing how to frame questions, identify problems, and make decisions; data-focused skillsknowing how to access, generate, and interpret data; and process-focused skillsknowing how to engage in collaborative inquiry and evaluate cause and effect (Mandinach & Gummer, 2013). Other data literacy skills include understanding the strengths and limitations in data collection and reporting tools (National Forum on Educational Statistics, 2012), as well as responding to concerns about equity by applying culturally responsive interventions and research-based instructional strategies to address achievement gaps identified in the data (Love, Stiles, Mundry, & DiRanna, 2008). After examining schools of education, data literacy experts, and state licensure requirements, Mandinach and Gummer (2015, this issue) propose the framework of data literacy for teaching as a comprehensive definition of data literacy, which combines traditional data analysis skills with content and pedagogical knowledge as educators use data to inform instruction.

Moreover, teachers are increasingly expected to use data literacy for teaching in collaborative inquiry as they work with other educators. Nelson and Slavit (2008) defined collaborative inquiry as a way of co-investigating a commonly agreed-upon element of teaching and learning (p. 103). Through teamwork, educators are better able to build data literacy for teaching: In a study of over 200 teachers, researchers discovered that educators were more comfortable and adept at interpreting data when working with groups of colleagues (Means et al., 2011). This emphasis on teamwork represents a departure from the way teachers have historically worked (Little, 1990; Lortie, 1975), but it is strongly supported by scholarship that argues that learning is built socially through dialogue and reflection with colleagues (Lave, 1996; Wenger, 1998; Vygotsky, 1978). In this perspective, teachers learn while interacting within communities that engage in critical and reflective discussions about instruction (Little, 1982, 2002; Putnam & Borko, 2000).

Collaborative work is also important because it makes teachers thinking the subject of discussion so that teachers can learn from one another. Hiebert, Gallimore, and Stigler (2002) described this as creating a knowledge base for teaching: Professional knowledge must be public, it must be represented in a form that enables it to be accumulated and shared with other members of the profession, and it must be continually verified and improved (p. 4). That is, professional knowledge must be based on context and concrete examples, integrated into teachers experiences, and then stored and shared with others. Effective collaborative work helps teachers engage in inquiry, or the stance of asking questions to shape conversation (McLaughlin & Talbert, 2002). Educators who engage in recursive cycles of collaborative inquiry are better able to understand the causal connections between the instructional practices they are using and student outcomes, ultimately resulting in higher student achievement (Gallimore, Ermeling, Saunders, & Goldenberg, 2009; Saunders, Goldenberg, & Gallimore, 2009).

New teachers must be better prepared to engage in collaborative data inquiry, but schools of education tend to include instruction about data in stand-alone courses rather than as an integrated approach to teaching, and they tend to focus mostly on test-based assessment data (Mandinach et al., 2015, this issue). Even within the limited category of assessment preparation, teacher candidates have the strongest preparation in finding different measures that track student learning, but teacher candidates are poorly prepared in responsibly analyzing that data to select instructional strategies within specific content or subject areas (Greenberg & Walsh, 2012). Teacher candidates are rarely taught to use data for whole-school improvement, as opposed to improving the performance of their own students, and they are rarely exposed to collaborative, team-based activities centered on data (Greenberg & Walsh, 2012). Overall, although schools of education might be building teachers assessment literacy, they are not preparing teachers to engage in data literacy for teaching, as defined earlier (Mandinach et al., 2015, this issue).

Other research on teachers practicing in schools further suggests that some veteran teachers do not have a solid foundation in data literacy or collaborative improvement skills. Indeed, many professional development experiences about data use in education focus on teaching data literacy in isolation rather than in support of instructional actions for school improvement (Mandinach & Gummer, 2013). There is evidence that educators go through a developmental progression in their understanding of using data for improvement, beginning with using data to identify students in need for specialized programs or services; then using data to make decisions about curriculum and whether to review material; and finally, looking at data as a way to design their pedagogy based on evidence of student learning (Means, Padilla, & Gallagher, 2010). For example, in a study of nine schools in two districts, teachers were more likely to use the results of interim assessment data primarily to decide what content to review with students, how to place students into instructional groups, and how to identify which students needed extra support, rather than rethinking how they would teach differently (Goertz, Oláh, & Riggan, 2009). Clearly, more needs to be done to support both novice and veteran educators in using data effectively. But what principles should guide the design of that support? In hopes of generating a discussion around this important question, the approach the Data Wise Project takes to working with practitioners and graduate students is described next.

THE DATA WISE IMPROVEMENT PROCESS

For over a decade, the Data Wise Project at the Harvard Graduate School of Education (HGSE) has been honing and teaching a model that provides a systematic approach to improving classroom practice. The work began when education economist Richard Murnane convened a group comprising researchers from HGSE and school leaders from three Boston public schools to work together to articulate what school leaders need to know and do in order to use data to improve instruction. The findings were captured in Data Wise: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning (Boudett, City, & Murnane, 2013). Contributions from Murnane, statistician John Willett, and psychometrician Daniel Koretz grounded the work in theory; contributions from practitioners ensured that it would be accessible and relevant to principals and teachers.

Figure 1 shows the Data Wise Improvement Process, which breaks the work of improvement into three phases. The prepare phase involves creating and maintaining a culture in which staff members can collaborate effectively and use data responsibly. In the inquire phase, educators use a wide range of data sources, including student work and classroom observations, to articulate a very specific problem of practice that they are committed to solving. In the act phase, teams articulate how they will learn about and employ instructional strategies to address this problem and how they will assess the extent to which the plan improved student learning. The model is characterized by an arrow that curves back on itself because after educators assess the effectiveness of their actions, they are well positioned to determine the focus for the next cycle of collaborative inquiry.

As they work their way through the process, teachers use field-tested protocols to examine a wide range of data sources. They then develop action plans that contain focused strategies for improving their teaching practice. These plans almost inevitably involve giving students a more central role in driving their own learning. Because teachers develop the plans themselves, their commitment to enacting them is much stronger than it would have been if central office or even school leaders handed down the plans. Because Data Wise explicitly supports and empowers teachers, it can serve as a useful counterbalance to the anxiety produced by increasing accountability pressures.

Figure 1. The Data Wise Improvement Process

Source: Boudett, K. P., City, E. A., & Murnane, R. J. (2013). Data Wise: A step-by-step guide to using assessment results to improve learning and teaching (revised and expanded ed.). Cambridge, MA: Harvard Education Press.

The Data Wise Project was established to support educators in using this process to organize the core work of schools around evidence of learning. Over the past 10 years, the Project has taught more than 2,500 educators worldwide who have enrolled in courses designed to either introduce the process or coach teams of educators in working their way through it. Most courses are designed for school- and district-based teams, although one is geared specifically to students in graduate programs. Delivery models range from programs that are entirely face-to-face to those that are completely online, including a course that employs a hybrid strategy. There are two different types of online courses: one that provides live custom coaching to each team and another that involves asynchronous communication between coaches and participating teams, with a particular focus on encouraging feedback among teams. Table 1 shows the purpose, audience, and delivery models for Data Wise courses.

Table 1. Summary of Data Wise Courses

	Introductory Course	Introductory Institute	Integration Courses
Participants	Graduate students in degree programs	School-level and district-level teams	School-level and district-level teams
Purpose	Prepare participants to get ready to lead collaborative inquiry	Prepare participants to get ready to lead collaborative inquiry and provide guidance as the journey unfolds	Provide participants with step-by-step coaching as their collaborative inquiry journey unfolds
Delivery Model	On Campus Only: One intensive week on campus at Harvard	Hybrid: One intensive week on campus at Harvard followed by three live online sessions	Online Only: One option involves live custom coaching over a period of 6 months; another involves asynchronous coach and peer support over 3 months

THREE HABITS OF MIND FOR USING DATA WISELY

Although all these courses use the Data Wise Improvement Process at the lead framework, participants are told from the start that simply checking off the steps of an improvement process will not by itself be enough to bring about real changes in learning and teaching. For meaningful change to occur, educators must bring a distinctive approach to their work. The Data Wise Project captured this disciplined way of thinking in the ACE Habits of Mind, in which each letter represents a habit:

A: Shared commitment to Action, Assessment, and Adjustment

C: Intentional Collaboration

E: Relentless focus on Evidence.³

Costa and Kallick (2008) defined habits of mind as what successful people do when they are confronted with problems to solve, decisions to make, creative ideas to generate, and ambiguities to clarify (p. 1). The ACE Habits of Mind are the stances that educators take when they approach data inquiry and improvement because they are ways of thinking that will help them achieve more productive outcomes. Adding an emphasis on habits of mind expands building data literacy beyond merely accumulating discrete knowledge and skills or learning a process that becomes routineit makes explicit that there are certain ways of thinking about data that separate emerging forms of data literacy from sophisticated, well-developed approaches.

Costa and Kallick (2008) noted that to activate habits of mind, one must be able to know the context in which such habits are useful and also have the capability and commitment to enact those habits. Because the ACE Habits of Mind do not come naturally for many educators, the instructional design of Data Wise courses provides opportunities that allow educators to experience the habits firsthand and incorporate them into their professional practice. Table 2 summarizes the teaching strategies that support the cultivation of each habit; however, it is important to note that using any one or even a few strategies will not help educators acquire the habits of mind automatically. The key is to explicitly teach the habits and then create learning experiences that incorporate the habits holistically, such that participants have practice activating the habits over time. The following sections discuss each set of strategies in turn. These strategies can be incorporated into any course about improvement, not just those that use the Data Wise Improvement Process specifically. Taken together, these strategies contribute to a robust instructional design regardless of the particular inquiry model used.

Table 2. Strategies for Supporting Educators to Cultivate Habits of Mind

Habit of Mind	Pedagogical Strategy
A Shared commitment to Action, Assessment, & Adjustment	" Organizing syllabus around actions that result from inquiry process steps " Teaching and modeling tools for gathering immediate feedback " Supporting thorough documentation of accomplishments and reflection
C Intentional Collaboration	" Requiring students to participate in the course as a team " Teaching and modeling tools for designing and facilitating effective meetings " Teaching and modeling tools for working in groups
E Relentless focus on Evidence	" Using case studies to expose participants to a broad range of data " Teaching and modeling tools for sticking to evidence " Providing structured opportunities for participants to discuss one anothers evidence

A: SHARED COMMITMENT TO ACTION, ASSESSMENT, AND ADJUSTMENT

The first of habit of mind that educators need to develop as they learn how to engage in collaborative data inquiry is an orientation toward purposeful, reflective action. When practiced broadly, this habit leads to organizational learning by creating a continuous feedback loop that allows educators to improve how they improve. The importance of feedback loops has been documented by organizational theorists, who argue that organizational learning occurs when cycles of action and reflection are iterative and recursive and build on one another over time (Argyris & Schön, 1996; Edmondson, 2002). Effective teams engage in reflection, or discussing new insights and learning, as well as action, or testing and implementing new ideas, applying insight, and producing change (Edmondson, 2002).

Unfortunately, educators have several bad habits that work against this habit of mind. Educators do not seek to practice bad habits, but the fragmented and rushed nature of much data work done in schools leads to bad habits that are created to manage the work. One such habit is the tendency to create action plans without following up on the results of those actions, without making midcourse corrections, and without reflecting on progress and learning over time (Boudett & City, 2013, p. 1). Educators are frequently asked to develop action plans for various reasons; for example, leadership teams create action plans focused on improving their schools performance on state assessments, and many educator evaluation systems require individual teachers to create action plans detailing how they will seek new professional development. However, they rarely have time to thoughtfully reflect on the results of those action plans in a way that produces learning and informs the next plan.

To be effective with data inquiry, educators need to complete the feedback loop and frequently gather evidence of progress and reflect on what happened as a result of their actions. There are three pedagogical strategies that institutions of higher education can use to help educators learn to choose this path.

Organizing the Syllabus Around Actions That Result From Inquiry Process Steps

Every class is explicitly tied to a step of the Data Wise Improvement Process, and course participants are often nervous that act is the very last phase. It is a misconception that the prior work done in the prepare and inquire phases does not involve action, because inquiry teams generate outcomes all along the inquiry cycle. The beginning of each class explains how the featured step of the improvement process connects to the ones before it and clarifies what the outcome of the step is that participants are working toward. This emphasis on intermediate outcomes helps participants see that action is required at every step. For example, when organizing for collaborative work (Step 1), the objective is to put into place the teams and structures that will support inquiry; when digging into student data (Step 4), the objective is to articulate a statement of the learner-centered problem that is supported by the data.

Teaching and Modeling Tools for Gathering Immediate Feedback

Courses place a heavy emphasis on training educators to pause and take stock of what they have learned. Discussion protocols, or rules for structuring conversations that help ensure a clear focus and broad engagement by everyone involved, often help teachers synthesize and reflect on learning (McDonald, Mohr, Dichter, & McDonald, 2007). Teams that use protocols or other structures to explicitly guide conversations tend to open up more opportunities for teachers to learn and participate (Levine & Marcus, 2010). One protocol⁴ that targets gathering immediate feedback is called plus/delta. It helps a group develop a shared sense of responsibility for improving by engaging everyone in assessing what worked well about a meeting, event, or plan, and what they would have liked to change. For example, by capturing plus/delta reflections within a meeting, it offers facilitatorsand participantsimmediate feedback on how to improve subsequent meetings. Another useful protocol called success analysis gives participants an opportunity to reflect deeply on a years work and articulate the specific things they did that contributed to improvement. The third protocol, called SUMI, allows participants to take a close look at the impact of protocols themselves. This protocol encourages participants to consider what Surprised them about a particular protocol, how they might Use or Modify it, and what Impact they think it might have. By engaging participants in a SUMI reflection, educators can understand that the primary reason for practicing a protocol during a class or session is to get them ready to use it in their professional work.

Requiring Thorough Documentation of Accomplishments and Reflection

To help educator teams reflect on their efforts with an improvement process, teams document their progress and learning over time in a journey presentation. For each step of the improvement process, teams create a summary that explains their process, evidence generated, and reflections on what was learned. By telling not just the story of what they did to prepare, inquire, and act, but also the story of any changes to teaching and learning, participants position themselves well to begin the cycle again with a new guiding question or a new way of thinking about the current issue. This written record can also serve as a tool for showing new team members what has happened so far and for demonstrating to other audiences at or beyond their school what the team has done.

C: INTENTIONAL COLLABORATION

This second habit of mind requires making deliberate decisions about how to engage colleagues in working together. Data inquiry relies on collaborative process: a team of educators sitting together discussing some form of data, whether it is aggregated student assessment scores or a classroom sample of completed student work. Educators who are able to enact this habit are thoughtful about which colleagues need to be part of the team, and they design and facilitate meetings so that each team member has opportunities to contribute to the data inquiry. The potential bad habit is assuming that just because a group of educators has been asked to work as a team, they will be collaborative and productive. Many have argued that team formation is not obvious or easyinstead, it requires leadership, routine processes, meaningful and interdependent tasks, and norms like trust, safety, and mutual accountability for completing work (Edmondson, 1999; Hackman, 2002; Katzenbach & Smith, 2003, Troen & Boles, 2012).

Collaboration is complex work, and many educators lack meeting facilitation and team leadership skills. Effective team facilitation is crucial to develop an inquiry-based culture; facilitators help to implement tools such as meeting agendas and discussion protocols that give routine and structure to collaboration (Gallimore et al., 2009; Nelson & Slavit, 2008; Park & Datnow, 2009; Talbert, Mileva, Chen, Cor, & McLaughlin, 2010). In a meeting where educators are engaged in data inquiry, facilitation enables all team members to contribute to the knowledge that is being created. Unfortunately, teacher preparation programs rarely teach educators how to be purposeful team facilitators, and thus new teachers enter the profession with little experience or knowledge about how to lead or support collaborative dialogue.

To help educators become more fluent collaborators, the following strategies are useful.

Requiring Students to Participate in the Course as a Team

In most of our courses, this means that everyone enrolls as a member of a school-level team (comprising the principal, a teacher, and usually at least one other faculty member) or a district-level team (comprising the people who are responsible for supporting schools in using data effectively). In the degree-program course, individual enrollees are placed on teams for the duration of the course and given frequent opportunities to practice their collaboration skills by working on a group project and reflecting on the experience. Having all enrollees engage as members of a team instead of as individuals provides them with a safe space to practice how to both facilitate and participate in collaborative meetings.

Teaching and Modeling Tools for Designing and Facilitating Effective Meetings

To address the lack of facilitation skills among educators, courses offer the Meeting Wise Checklist as a tool for planning and evaluating meetings (Boudett & City, 2014). The checklist encourages participants to think about the purpose, process, preparation, and pacing of their meetings. It places particular emphasis on articulating meeting objectives, assigning roles to participants (such as facilitator, timekeeper, and note taker), and planning and following up on next steps. Instructors also build participants capacity by practicing transparent facilitation (the equivalent of the instructional think aloud) while working with school teams. For example, facilitators might explain why they decided to shorten a protocol because of time constraints or why they chose to write suggestions on chart paper to keep a visual record of what was said in conversation. By making the decision-making process that facilitators use explicit, this practice encourages participants to notice this often invisible skill.

Teaching and Modeling Tools for Working in Groups

It is important to set the expectation early on that every participant can grow as a facilitator, and to use protocols and agendas to take the mystery out of facilitation by providing a sequence of steps and time limits for conversations. These are valuable tools for supporting the habit of intentional collaboration because they make it clear when each person should contribute ideasand when he or she should to listen to the ideas of others. At the same time, there is a need to clarify that each individual will bring his or her own style and preferences to the teams experience. To help participants to understand this, courses often begin by engaging participants in the Compass Points protocol, which helps them understand how team members best learn and participate in group work.⁵ For example, some team members may want to move quickly to action or creating products, whereas other team members would prefer to discuss and ensure that everyone has a say in the process. Once team members have discussed these preferences, they are often more comfortable setting norms or expectations for how their team will work together. When informed by conversations about group preferences, the norms are more authentic, rather than simply rules for behavior that have little grounding in an understanding of how a particular team prefers to operate. For example, although common norms specify that the meetings should assume positive intentions, this norm becomes more important after team members have discussed how people who seek the big pictureas well as those who are determined to get right down to detailsare acting from their sincere instincts about what will help the team make progress toward its goals rather than trying to be oppositional.

E: RELENTLESS FOCUS ON EVIDENCE

This habit is closely linked to the habit of intentional collaboration, and educators frequently report that the single most important thing that builds trust in collaboration is making sure they ground statements in evidence. This habit supports a culture in which people make decisions based on specific, objective, and descriptive statements about what they see. But drawing conclusions from data that are unfounded or unwarranted is a deeply ingrained bad habit for most people, so it takes a determined effort to break it. It is easy to fall into this bad habit because human beings are naturally predisposed to making meaning and creating interpretations using data from the world around us (Senge et al., 2000). However, these inferences or judgments are not helpful at the early stages of data inquiry because they move educators away from looking closely at the data and honing the practice of observation. As an example, a statement such as this student doesnt care about school is more likely to close down possibilities for inquiry and prevent conversations about alternative explanations than a factual statement like this student has not turned in assignments for three days.

Another bad habit that might arise is when educators have a narrow definition of what constitutes useful data. Many people begin by assuming that data refers to numeric representations of student performance on assessments, such as how many students in a grade obtained scores below, at, and above levels of proficiency on a high-stakes standardized test. The accountability system in the United States, which allocates punishments or rewards based on aggregates of student performance, exerts pressures on educators to use data in certain ways (Jennings, 2012). At times, the accountability pressures are so strong that they might dominate educators initial assumptions about what forms of data are most important to examine, and school teams arrive at Data Wise expecting to learn how to increase student performance on standardized tests. Fortunately, education as a field has become wiser about the need to triangulate, or examine multiple forms of data that provide evidence of student learning (Love et al., 2008; Patton, 2002). Further, assessment experts have reinforced the idea that consequential decisions about student learning should not be based on the results of a single test (Koretz, 2009). Researchers document instances where data teams meet to examine not only test scores but student work produced in class and for homework, and they argue that the most helpful sources of instructional information are assessments that demonstrate student thinking, developmental pathways or approaches, or misconceptions (Supovitz, 2012).

Practitioners often overlook data collected from classroom observations about student learning and teaching practice. Yet, being able to accurately describe what is currently happening in classrooms is essential to figuring out what improvements to instruction are needed (City, Elmore, Fiarman, & Teitel, 2009). When the data in question are observations about practice, it is particularly important to cultivate the habit of maintaining a relentless focus on evidence. When an educator reports that teachers in multiple classrooms failed to engage students in rigorous work, a skillful facilitator will ask that person to provide the evidence supporting his or her claim. A more useful data statement about classroom practice might be: Most students answered teachers questions with one- or two-word statements.

Strategies for helping educators develop a relentless focus on evidence follow.

Using Case Studies to Expose Participants to a Broad Range of Data

Research supports the idea that children learn best when abstract ideas are put into a context that is meaningful to them (National Research Council, 2000); the same is true for adults (Kolb & Kolb, 2005). Efforts to teach educators how to read score reports or develop action plans can fall flat if not embedded in case studies of real schools. Teaching educators data literacy through examples of student work or videos of classroom instruction from real schools helps them learn new skills while putting themselves in the shoes of the educators in the cases. It also increases engagement by making it easier for participants to see how to transfer their learning to their own situations. Finally, case studies demonstrate how real educators gather a broad range of data to address one focus of inquiry. By following a case study school throughout the entire inquiry cycle, participants see firsthand how a school or educator team might start by examining student performance data, then move to looking at student work, and finally to observing classroom instruction. Each next source of data adds another layer of understanding about students and teachers strengths and challenges.

Teaching and Modeling Tools for Sticking to Evidence

Once again, protocols can be quite useful for helping educators try out habits that might at first feel unfamiliar. For example, many educators think that when they observe a classroom, it is their job to identify what the teacher is doing right and wrong. So when they observe a video of instruction, participants should take very descriptive and specific notes about what they see and hear students and teachers doing. The affinity protocol helps educator teams analyze the classroom observation data. This protocol allows educators to work with colleagues to sort, categorize, and label this observational evidence and come to a shared understanding of what is happeningand not happeningin classrooms. In addition, educators might use the mental model of the ladder of inference, an idea developed by Chris Argyris, Peter Senge, and others (Argyris & Schön, 1996; Senge et al., 2000). The bottom rung of the ladder contains descriptive statements, but as one climbs the ladder conceptually, the higher rungs lead to inferences, conclusions, and actions. Participants learn how to classify their statements and how to go up the ladder deliberately, making sure they have enough evidence to support the climb. To avoid making judgmental statements, participants might ask one another, What evidence do you see that makes you say that? The metaphor of being high on the ladder and coming down the ladder can make the process of developing the habit of evidence more playful and easier to talk about and do.

Providing Structured Opportunities for Participants to Discuss One Anothers Evidence

Perhaps the most effective strategy to maintain a relentless focus on evidence is having educators share the evolving evidence of their learning with their classmates. Students in the week-long degree program course engage in peer consultancies in which they present and get feedback on their group projects. Teams enrolled in the professional education courses, which extend over a period of months, have even richer opportunities for discussing ongoing work. After learning the process during an intense week on Harvards campus, teams return to their settings, where they integrate the improvement process into their daily work and document what happens. During virtual coaching calls and live online sessions with their institute peers, teams practice marshaling evidence of their improvement work, describing it to others, and listening to their peers use evidence to describe their successes and challenges. In addition to building participants muscles around using evidence to support their claims, this instructional approach helps sustain motivation as participants share work, receive feedback, make revisions, and hear from colleagues.

TRANSLATING LESSONS FROM THE FIELD

On the last day of a Data Wise course, it is typical for educators to report that the experience was not what they expected. One year, a student summarized her learning by writing, I used to think that it was all about the data: its accuracy, validity, the amount we have. And now I think that to achieve success in using data to affect change, the attitudes and skills of the people implementing the change are more important. Interestingly, in the early years, our courses placed more emphasis on helping educators to retrieve assessment results from databases and create formal presentations of their data. However, feedback from participants revealed that they needed more than a toolkit of discrete skills, protocols, and formsthey needed a way to understand the process of inquiry in its entirety and the habits of mind that they could integrate across all their work with data.

Reciprocal relationships between institutions of higher education, professional developers, and K12 educators are essential to supporting educators to become data literate. As noted in prior research, very few teacher candidates learn to engage deeply in data literacy for teaching in their coursework. Further, even when teacher candidates are placed in schools with a mentor teacher who can model and support the development of good instructional practices, there is no guarantee that the novice teacher will be exposed to a team that engages in data inquiry. Novice teachers are socialized into the professional act of collaborative improvement vicariously, if at all. As a result, we arrive at our current situation: Most educators have not learned data literacy skills in preservice preparation, such that schools and districts must provide ongoing professional development to make up for this gap.

Institutions of higher education and professional development providers must encourage learning through clinical practice, meaning courses that combine practitioners and students occurring in real school settings and using authentic examples. It is usually said that institutions of higher education have a responsibility to bridge what is known from research studies with the everyday work of educators. The Data Wise Project sees its responsibility as both bringing research to practitioners and translating knowledge from practitioners into frameworks, tools, and supports for other educators. This involves creating instructional designs and pedagogy that are responsive to participants needs. Practitioners are therefore involved in coauthoring materials, codesigning educational experiences, and cofacilitating our courses. Their reflections and journey presentations demonstrate how educators apply the Data Wise Improvement Process at their schools, including how they have integrated the process with their other work. Practitioners questions inform the next iteration of instructional design; their frustrations lead to the creation of new practical tools and problem-solving suggestions, and their successes are documented in ways that can be shared for the benefit of others learning.

CONCLUDING THOUGHTS

Education has been assailed for lacking ways to socialize educators into the profession, leading many to observe that teaching practice in schools appears haphazard and largely attributable to each individual teachers dispositions. In his call for defining and developing signature pedagogies in education, Shulman (2005) described a vision that education might have a set of practices to induct novices into ways of thinking like an educator, and all institutions and programs that train and support educators would use these methods. These signature pedagogies would encourage the development of habits of the mind, habits of the heart, and habits of the hand common to all members in the profession (p. 59). As the field continues in search of a signature pedagogy that might socialize new teachers into a collaborative, data-literate profession, there must be less emphasis on teaching data literacy as a set of isolated skills and practices, and instead a focus on teaching data inquiry as a holistic process grounded in habits of mind necessary for collaborative improvement. The goal is not just getting teachers to be comfortable with data, but allowing the profession to evolve to a place where understanding of data is thoroughly integrated with the work of learning and teaching.

Notes

1. In this article, we refer to educators instead of just teachers to include administrators, specialists, and coaches.

2. For more information about the Data Wise Project, please visit http://www.gse.harvard.edu/datawise.

3. For further discussion of the ACE Habits of Mind, see Boudett & City, 2013.

4. Instructions for all the protocols mentioned in this article can be found online: Plus/Delta, Affinity, SUMI: http://www.gse.harvard.edu/datawise; Success Analysis: http://www.tcpress.com/pdfs/mcdonaldprot.pdf; Compass Points: http://www.schoolreforminitiative.org/.

5. For instructions, please see http://www.schoolreforminitiative.org/.

References

Argyris, C., & Schön, D. A. (1996). Organizational learning II: Theory, methods and practice. Reading, MA: Addison-Wesley.

Boudett, K. P., & City, E. A. (2013). Lessons from the Data Wise Project: Three habits of mind for building a collaborative culture. Harvard Education Letter, 29(3), 12. Retrieved from http://hepg.org/hel/article/567

Boudett, K. P., & City, E. A. (2014). Meeting wise: Making the most of collaborative time for educators. Cambridge, MA: Harvard Education Press.

Boudett. K. P., City, E. A., & Murnane, R. J. (Eds.). (2013). Data Wise, revised and expanded edition: A step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard Education Press.

City, E. A., Elmore, R. F., Fiarman, S. E., & Teitel, L. (2009). Instructional rounds in education: A network approach to improving teaching and learning. Cambridge, MA: Harvard Education Press.

Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement, 9, 173206.

Coburn, C. E., & Turner, E. O. (2012). The practice of data use: An introduction. American Journal of Education, 118(2), 99111.

Costa, A. L., & Kallick, B. (Eds.). (2008). Learning and leading with habits of mind: 16 essential characteristics for success. Alexandria, VA: Association for Supervision and Curriculum Development.

Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(1999), 350383.

Edmondson, A. (2002). The local and variegated nature of learning in organizations: A group-level perspective. Organization Science, 13(2), 128146.

Gallimore, R., Ermeling, B., Saunders, W., & Goldenberg, C. (2009). Moving the learning of teaching closer to practice: Teacher education implications of school-based inquiry teams. Elementary School Journal, 109(5), 537553.

Goertz, M., Oláh, L., & Riggan, M. (2009, December). Can interim assessments by used for

instructional change? (CPRE Policy Briefs RB-51). Philadelphia, PA: Consortium for Policy Research in Education.

Greenberg, J., & Walsh, K. (2012). What teacher preparation programs teach about K-12 assessment: A review. Washington, DC: National Council on Teacher Quality. Retrieved from http://www.nctq.org/p/publications/docs/assessment_report.pdf

Hackman, R. (2002). Leading teams: Setting the stage for great performances. Cambridge, MA: Harvard Business School Press.

Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009).

Using student achievement data to support instructional decision making (NCEE 2009-4067). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides/

Hiebert, J., Gallimore, R., & Stigler, J. (2002). A knowledge base for the teaching profession:

What would it look like and how would we get one? Educational Researcher, 31(5), 315.

Jennings, J. L. (2012). The effects of accountability system design on teachers use of test
score data. Teachers College Record, 114(11), 123.

Katzenbach, J. R., & Smith, D. K. (2003). The wisdom of teams: Creating the high performance

organization. Cambridge, MA: Harvard Business School Press.

Kekahio, W., & Baker, M. (2013). Five steps for structuring data-informed conversations and

action in education (REL 2013001). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Pacific. Retrieved from http://ies.ed.gov/ncee/edlabs

Kolb, A. Y., & Kolb, D.A. (2005). Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of Management Learning and Education, 4(2), 193212.

Koretz, D. (2009). Measuring up: What educational testing really tells us. Cambridge, MA: Harvard University Press.

Lave, J. (1996). The practice of learning. In S. Chaiklin & J. Lave (Eds.), Understanding practice: Perspectives on activity and context (pp. 332). Cambridge, England: Cambridge University Press.

Levine, T. H., & Marcus, A. S. (2010) How the structure and focus of teachers collaborative activities facilitate and constrain teacher learning. Teaching and Teacher Education, 26(3), 389398.

Little, J. W. (1982). Norms of collegiality and experimentation: Workplace conditions of school success. American Educational Research Journal, 19, 325340.

Little, J. W. (1990). The persistence of privacy: Autonomy and initiative in teachers professional relations. Teachers College Record, 91(4), 509536.

Little, J. W. (2002). Locating learning in teachers communities of practice: Opening up

problems of analysis in records of everyday work. Teaching and Teacher Education, 18(7), 917946.

Lortie, D. C. (1975). Schoolteacher: A sociological study. Chicago, IL: University of Chicago.

Love, N., Stiles, K. E., Mundry, S., & DiRanna, K. (2008). The data coachs guide to improving learning for all students: Unleashing the power of collaborative inquiry. Thousand Oaks, CA: Corwin Press.

Mandinach, E. B., Friedman, J. M., & Gummer, E. S. (2015). How can schools of education help to build educators capacity to use data? A systemic view of the issue. Teachers College Record, 117(4).

Mandinach, E. B., & Gummer, E. S. (2013). Defining data literacy: A report on a convening of experts. Journal of Educational Research and Policy Studies, 13(2), 628.

Marsh, J. A. (2012). Interventions promoting educators use of data: Research insights and gaps. Teachers College Record, 114(11), 148.

McDonald, J. P., Mohr, N., Dichter, A., & McDonald, E. C. (2007). The power of protocols (2nd ed.). New York, NY: Teachers College Press.

McLaughlin, M., & Talbert, J. E. (2002). Reforming districts: How districts support school reform. Seattle: Center for the Study of Teaching and Policy, University of Washington.

Means, B., Chen, E., DeBarger, A., & Padilla, C. (2011). Teachers' ability to use data to inform instruction: Challenges and supports. Washington, DC: U.S. Department of Education Office of Planning, Evaluation and Policy Development.

Means, B., Padilla, C., & Gallagher, L. (2010). Use of education data at the local level: From accountability to instructional improvement. Washington, DC: U.S. Department of Education Office of Planning, Evaluation and Policy Development.

National Forum on Education Statistics. (2012). Forum guide to taking action with education data (NFES 2013-801). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

National Research Council. (2000). How people learn: Brain, mind, experience, and school (expanded ed.). Washington, DC: National Academies Press.

Nelson, T., & Slavit, D. (2008). Supported teacher collaborative inquiry. Teacher Education Quarterly, 35(1), 99116.

Park, V., & Datnow, A. (2009): Co-constructing distributed leadership: district and school connections in data-driven decision-making. School Leadership & Management, 29(5), 477494.

Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.

Putnam, R. T., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29(1), 415.

Saunders, W. M., Goldenberg, C. N., & Gallimore, R. (2009). Increasing achievement by focusing grade-level teams on improving classroom learning: A prospective, quasi-experimental study of Title I Schools. American Educational Research Journal, 46(4), 10061033.

Senge, P., Cambron-McCabe, N., Lucas, T., Smith, B., Dutton, J., & Kleiner, A. (2000). Schools that learn: A Fifth Discipline fieldbook for educators, parents, and everyone who cares about education. New York, NY: Crown Business.

Shulman, L. S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 5259.

Supovitz, J. (2012). Getting at student understandingThe key to teachers use of test data. Teachers College Record, 114(11), 129.

Supovitz, J., Foley, E., & Mishook, J. (2012) In search of leading indicators in education. Education Policy Analysis Archives, 20(19) Retrieved from http://epaa.asu.edu/ojs/article/view/952

Talbert, J. E., Mileva, L., Chen, P., Cor, M. K., & McLaughlin, M. (2010). Developing school 

capacity for inquiry­based improvement: Progress, challenges, and resources. Stanford, CA: Center for Research on the Context of Teaching.

Troen, V., & Boles, K. C. (2012). The power of teacher teams: With cases, analyses, and strategies for success. Thousand Oakes, CA: Corwin.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (M. Cole, V. John-Steiner, S. Scribner, & E. Souberman, Eds.). Cambridge, MA: Harvard University Press.

Wayman , J. C., Jimerson, J. B., & Cho, V. (2012). Organizational considerations in establishing the Data-Informed District. School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice, 23(2), 159178.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge, England: Cambridge University Press.

Young, V. M., & Kim, D. H. (2010). Using assessments for instructional improvement: A literature review. Education Policy Analysis Archives, 18(19), 140.

Cite This Article as: Teachers College Record Volume 117 Number 4, 2015, p. 1-20 https://www.tcrecord.org ID Number: 17853, Date Accessed: 10/23/2021 11:00:24 AM Purchase Reprint Rights for this article or review