Leveraging Observation Tools for Instructional Improvement: Exploring Variability in Uptake of Ambitious Instructional Practices
by Julie Cohen, Lorien Chambers Schuldt, Lindsay Brown & Pamela Grossman - 2016
Background/Context: Current efforts to build rigorous teacher evaluation systems has increased interest in standardized classroom observation tools as reliable measures for assessing teaching. However, many argue these instruments can also be used to effect change in classroom practice. This study investigates a model of professional development (PD) built around a tool—the Protocol for Language Arts Teaching Observations (PLATO).
Purpose/Objective: The study analyzes the extent to which teachers appropriated the instructional practices targeted in the PLATO PD. We also assess factors that may have supported and/or hindered teachers’ uptake of practices.
Setting/Participants: The study sample includes 27 teachers who participated in PD over 2 years. Teachers worked in six middle schools in a single, large urban district.
Intervention: The two year PD consisted of 5 daylong sessions each school year, and a 4-day summer institute. All sessions focused on the PLATO scales. Teachers also worked in school-based teams to design lessons featuring the focal practices and attended five school-site meetings with PLATO PD providers.
Research Design and Data Collection: PLATO served as a set of practices around which to orient PD, as well as a standardized tool for measuring changes in teacher practice. All teachers were observed using PLATO scales throughout the PD and during the subsequent year. We conducted multiple interviews with all participating teachers, which were transcribed and coded by multiple researchers. Case studies of six purposively sampled teachers incorporate interviews, scores, and field notes. Findings/Results: The duration of PD mattered in terms of teachers’ appropriation of PLATO practices. In addition, “foundational practices” supported the appropriation of more ambitious practices targeted in the PLATO PD, including time and behavior management and instructional planning. Finally, our findings suggest stable and collaborative communities support professional learning and growth.
Conclusions/Recommendations: The findings suggest moving away from “one size fits all models” and differentiating PD for teachers. Effective professional development may not be effective for all teachers. Observation protocols can play a unique role in PD by allowing professional developers to gather standardized information across teachers and to compare changes in teacher practice in systematic ways. PD providers might also use such tools diagnostically to identify and respond to the heterogeneity in teachers’ practice.
INTRODUCTION AND RESEARCH QUESTIONS
The current effort to build more rigorous teacher evaluation systems has increased interest in classroom observation tools, which are designed to be valid and reliable measures for assessing teaching performance (Gitomer, 2008; Kane & Staiger, 2012). Many argue that these same instruments can also serve as a lever to effect change in classroom practice, although we have limited empirical evidence to bolster such claims (Allen, Pianta, Gregory, Miakmi, & Lun, 2011; Papay, 2012). This study investigates a model of professional development built around such a toolthe Protocol for Language Arts Teaching Observations (PLATO)that was designed to measure the quality of teaching in middle school English Language Arts (ELA) classrooms, based on research on ELA instruction and adolescent literacy. Several studies have demonstrated significant relationships between scores on the practices assessed by PLATO and student achievement gains (Grossman, Brown, Chambers Schuldt, & Cohen, 2014; Grossman, Brown, Chambers Schuldt, Metz, & Johnson, 2013; Kane & Staiger, 2012).
The goal of this project was to use PLATO, a tool we helped develop, as the basis for professional development aimed at instructional improvement. Our model of professional development involved introducing teachers to the PLATO rubricrevised for professional development purposesand then targeting several PLATO practices for intensive professional development. Participating teachers selected two instructional practices, facilitating Classroom Discourse and Strategy Use and Instruction, to focus on over the course of the 2-year professional development program. Previous research has shown the quality of practice in these two areas is relatively weak, and that these practices are associated with student achievement gains (Grossman et al., 2014; Grossman et al., 2013; Kane & Staiger, 2012). This study explores the extent to which teachers improved in their enactment of these practices over the course of the professional development experience, as well as identifying factors that may have supported or hindered teachers uptake of these particular practices.
Although a growing body of work has helped to identify features of effective professional development, most studies have relied on measures of main effects of a specific intervention (e.g., Borko, Jacobs, & Koellner, 2010; Desimone, 2009; Garet, Porter, Desimone, Birman, & Yoon, 2001; Wilson & Berne, 1999). Relatively little is known about either the heterogeneity that likely underlies these overall effects, or the mechanisms that support or impede change in teachers practice. Why do certain teachers appropriate the knowledge, skills, and practices highlighted in professional development while others do not? A critical next step to improving instruction is developing a more robust understanding of how to support teachers development of these ambitious teaching practices. Better specification of these processes, and the factors that may mediate the impact of professional development, could allow us to more precisely target sites and/or teachers for whom particular professional development experiences would likely be especially beneficial, and potentially ameliorate the factors impeding growth in teaching practice. As Hill, Beisiegel, and Jacob (2014) recently argued, the field will benefit from more systematic research about the specific features of professional development that contribute to effectiveness, with particular attention to the factors that may explain the heterogeneity of change in teachers practice.
This mixed methods study begins to address this need by exploring the factors that may contribute to the degree to which teachers appropriate a set of these PLATO practices during and after a 2-year, intensive professional development experience. Using both quantitative and qualitative methods, we begin building theory about what contributes to teachers differential uptake of these practices. The study addresses the following research questions:
To what extent do teachers appropriate the instructional practices targeted in the PLATO professional development?
What factors support and/or hinder teachers uptake of PLATO practices?
BACKGROUND AND FRAMEWORK
Changing teacher practice is difficult. Decades of research on professional development demonstrate inconsistent evidence about the impact of such investments in teacher development on teaching practices and student learning (Duffy et al., 1986; Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). That said, there is a growing body of research that identifies general features of effective professional development, including a focus on content-specific aspects of teaching (cf. Borko et al., 2010; Desimone, 2009; Garet et al., 2001). Time, for example, is also a necessary precondition for learning for students and teachers alike. Prior research suggests that duration is a crucial factor for successful professional development. More sustained professional development experiences are associated with a number of outcomes: self-reported increases in teachers knowledge and skills, changes in classroom practice, and greater effects on student achievement (Garet et al, 2001; Kennedy, 1998; Yoon et al., 2007). In addition, institutional resources, such as having multiple participants from a particular school context (Penuel, Fishman, Yamaguchi, & Gallagher, 2007; Sun, Penuel, Frank, Gallagher, & Young, 2013) and administrative support (Guskey, 2002; Little, 2003), appear to foster collaboration and promote enhanced teacher learning. Finally, studies suggest teachers benefit from opportunities to plan, enact, and discuss the principles underlying the practices featured in professional development programs, as well as from tight alignment between the perceived goals of the professional development and their personal, school, and district-level instructional goals (Penuel et al., 2007; Wilson & Berne, 1999).
However, even when professional development opportunities adhere to these recommendations and feature these kinds of opportunities, teachers show differential uptake of professional development content and practices (Lieberman & Wood, 2002; Whitney, 2008). For example, the program of research on Cognitively Guided Instruction (CGI)a professional development program focused on how elementary teachers can use student thinking to enhance elementary mathematics instructionhighlights the variability in how teachers appropriated the tools, terminology, and practices presented in the CGI professional development. In particular, Knapp and Peterson (1995) underscore the differences between those who adopted a CGI mindset and saw CGI as transforming both their beliefs and practices about teaching mathematics, and participants who saw CGI only as a set of procedures to supplement instruction. Franke and Kazemi (2001) similarly noted that some, but not all, CGI participants used the professional development as a generative learning experience in that practice serve[d] as a basis for continued growth and teacher experimentation and reflection (p. 68).
The existing literature provides little detail about the factors that contribute to more generative and transformative professional development experiences for teachers. As a result, professional development providers and districts have little insight into why or how specific teachers seem to benefit more or less from the professional development in which they participate. This study focuses squarely on these differences in how teachers engage with and capitalize on professional development around teaching practices. In particular, we draw on theory around the appropriation of tools and practices for teaching (Grossman, Smagorinsky, & Valencia, 1999; Moscovich, 2013). In earlier work, we highlighted the difference between conceptual tools, or principles or ideas that guide decisions about teaching and learning across multiple content areas or types of lessons, and practical tools that that have more local and immediate utility (Grossman et al., 1999). Drawing on that work, the ultimate goal of the PLATO professional development was that participating teachers develop a flexible understanding of PLATO practices that would allow them to repurpose the practices as adaptive and generative tools across varied instances of instruction.
We focus here on understanding why and how some teachers took up the PLATO practices as different types of tools. We draw on quantitative data to explore teachers uptake of practice, along with qualitative data to explore the factors that may have shaped the extent to which teachers understood the practices as a set of practical tools (model lessons, specific texts etc.) to use in their classrooms, or conceptual tools to continually use in generative and flexible ways in their teaching. In this study, PLATO served as both a set of practices around which to orient professional development in English Language Arts instruction, as well as a standardized tool with which to measure changes in teacher practice.
DATA AND METHODS
The study sample includes 27 teachers who participated in PLATO professional development over the course of 2 years. Our sampling began by first identifying participants from middle schools in a single, large urban district in the western United States. The district identified a set of seven middle schools from which we could draw, and in the 1st year, four schools agreed to participate. Principals then helped identify 12 ELA teachers across the four schools. Although the study was voluntary, many teachers were strongly encouraged by their principals to attend.1 We refer to these 12 teachers as Cohort 1. One teacher left the project after Year 1, due to an out-of-district move.2
In Year 2, we recruited an additional 10 teachers from these four schools, and two additional schools joined the projectfor a total of 15 teachers in what we refer to as Cohort 2.3 Teachers were asked to complete a brief survey of their prior teaching experience, credentials, and demographic information such as age and race (see Table 1 for details of the sample).
Table 1. Sample of Teachers
Twenty-five of the 27 teachers taught at least two sections of ELA; some also taught social studies as part of a Humanities block. Teachers levels of experience ranged from 1 to 23 years of classroom teaching. The sample of teachers who participated in the study is relatively similar to the population of middle school ELA teachers in the district as a whole in terms of age and years of experience, though somewhat less ethnically and racially diverse (see Table 2). Additionally, teachers in the study sample had fewer years of teaching experience on average, and smaller percentages of teachers in the sample had masters degrees than the district middle school ELA teachers.
Note: Stars indicate statistically significant differences between the study sample and teachers in the district: * p < .05. ** p < .01. *** p< .001
THE PLATO TOOL
The PLATO rubric includes 13 teaching practices,4 such as Modeling, Connecting to Prior Academic Knowledge, and Classroom Discourse, that research suggests are important to student learning in English Language Arts (e.g., Beck & McKeown, 2002; Greenleaf, Schoepenhauer, Cziko, & Mueller, 2001; Hillocks, 1995; Langer & Applebee, 1986; Nystrand & Gamoran, 1991; OConnor & Michaels, 1993; Snow & Biancarosa, 2003; Sperling & Freedman, 2001). Exploratory and confirmatory factor analyses suggest that the 13 elements are clustered in four broader domains: Disciplinary Demand of Classroom Activity and Discourse, Instructional Scaffolding of ELA Content, Representations and Use of Content, and Classroom Environment (see Table 3).5 Each observation segment is independent and captures 15 minutes of instruction. Raters have 8 minutes to record scores before beginning a new 15-minute segment. Scores range from 1 (low) to 4 (high). Due to the wide range of ELA content captured by PLATO, it would be exceedingly rare for a teacher to obtain a high score on every element for any given 15-minute segment.
THE PLATO PROFESSIONAL DEVELOPMENT PROGRAM
In its entirety, the professional development lasted for 2 years, consisting of five full-day, off-site sessions each school year. The first cohort of teachers participated in the full 2 years of PD, while the second cohort joined in Year 2. Teachers in both cohorts attended a 4-day institute during the summer between Year One and Year Two. In total, teachers in Cohort 1 had 14 full days over the 2-year period, and teachers in Cohort 2 had 9 days of a year-long period. Professional development was oriented around the PLATO scales, beginning with an overview of the PLATO tool and then focusing on the two focal elements, Classroom Discourse and Strategy Use and Instruction for the majority of the sessions. During each day of professional development, teachers were presented with direct instruction about one or both of the focal practices, representations of the practice including videos and model lessons, and opportunities to rehearse the practice in structured activities. Teachers also worked in school-based teams to design lessons featuring the focal practices.
In addition to the off-site sessions, teachers were asked to attend five school-site meetings with PLATO participants at their school, which were facilitated by members of the research team. Teachers often had specific tasks for the meetings that were related to the instructional concept or practices emphasized in the most recent off-site sessions. At both on and off-site sessions, teachers were given time to share their work or work in progress with colleagues and ask for specific feedback.
Observations using the PLATO tool served as a key data source for the study. Each teacher was observed over the course of three cycles for each year of participation in the professional development, as well as for the subsequent year. We observed Cohort 1 teachers for 3 years total (two during the professional development and one after), and Cohort 2 teachers for 2 years (one during the professional development and one after). Teachers were observed an average of four times in each cycle, with a minimum of two and a maximum of eight lessons observed per cycle. The structure and frequency of observations was designed to maximize the likelihood of reliable estimates of teacher practice, based on earlier generalizability studies of the tool (Cor, 2011). Over the course of the study, teachers were observed for a total of 9 to 35 lessons; the variance is due to teachers participating in the study for different lengths of time, as well as constraints due to time of year, teacher availability, and school events such as standardized testing. Seven trained raters conducted the observations. All raters had ELA teaching experience and achieved at least 80% interrater agreement on each PLATO element with master-coded segments. Approximately 10% of observations were double-scored to ensure high levels of ongoing interrater agreement (> 80% exact-score agreement).
Qualitative data sources include: teacher interviews, transcripts of the professional development sessions and subsequent debriefs, and transcripts from the school-site meetings. These data provide additional evidence of the degree to which teachers understood the PLATO practices as conceptual or practical tools, as well as some factors to explain those differences. Members of the research team conducted interviews with all 27 teachers at multiple points across the study: the beginning of their participation in the professional development, during the spring of their first year, and then again during the fall after the professional development was complete. Cohort 1 participated in an additional interview during the spring of their second year. Interviews asked about respondents vision of good ELA instruction, as well as their description of ELA instruction, professional development, initiatives, policies, and resources at their school. The later interviews asked respondents to describe their understanding of the PLATO target practices, and to reflect on their experience participating in the professional development.
We also draw on transcripts of the professional development sessions and the debrief sessions that the providers engaged in after each professional development. These data provided contextual information about how individual teachers responded to the professional development, as well as the observations of the professional developers immediately following a session. Transcripts of audio recordings as well as the facilitator notes for periodic school-site meetings were used to analyze factors underlying teachers differential uptake of the PLATO practices.
PLATO observation scores. Growth in teachers scores on PLATO was measured by change in scores from the baseline cycle to the highest-scoring cycle, and in the rate of what we term throughout the rest of this paper as higher scoring practice as measured by percent of total observation segments that reached the 3 or 4 level in the two target practices of Classroom Discourse and Strategy Use and Instruction.
We also ran several analyses to investigate whether certain foundational skills are associated with appropriation of the PLATO instructional practices targeted in the professional development. For example, classroom management practices have been thought to be a precondition of instructional growth for teachers and students alike (Kagan, 1992), though this has been a contested topic amongst teacher educators (cf. Grossman, 1992). Indeed, the organizational ability and classroom management of teachers has long been shown to support student learning (Bastian, 2013; Good & Grouws, 1977; Mashburn et al., 2008). We might therefore predict that teachers who better manage time, spending fewer instructional minutes on transitions and behavior management issues, would be better able to implement the instructional practices introduced in professional development, and ultimately be better able to support student learning. Therefore, we ran models estimating both growth and higher scoring practice on the target instructional practices as a function of scores on the PLATO scales for Time and Behavior Management.
To investigate the relationship between increased participation in PD and teacher instructional change, an independent samples t test was conducted to compare growth on the targeted instructional practices between Cohorts 1 and 2. The t test was two tailed and did not assume equal variances between groups.
Finally, given research that suggests that teaching practices may be shaped by teachers prior experiences and professional preparation (Henry, Bastian, & Smith, 2012; Boyd, Grossman, Lankford, Loeb, & Wyckoff, 2009), we conducted a number of analyses that predict growth and higher scoring practice as a function of various teacher characteristics. These bivariate analyses regress the targeted PLATO variables, measured continuously, on teacher self-reported data regarding certification and prior teaching experience. Prior teaching experience variables are measured continuously in years, whereas certification variables are measured as binary variables.
Case Study Analysis. To complement the quantitative analyses and enrich our understanding of the heterogeneity of teachers uptake of PLATO practices, we identified a set of teachers to study in greater depth. Given our interest in factors that supported or hindered teachers uptake of the PLATO practices, we used a stratified, purposive sample to capture maximum variability in appropriation. We selected six case study teachers who demonstrated a range of growth and varying levels of sophistication of practice. Teachers were ranked according to the changes in practice (measured from baseline to highest scoring cycle and averaged for Classroom Discourse and Strategy Use and Instruction) and percent of higher scoring practice (average of Classroom Discourse and Strategy Use and Instruction). These two measures were then used to create a 3 ´ 3 sampling matrix, and teachers were divided into terciles (high, mid and low) for each measure (Yin, 2003). Two teachers were selected from each of the high/high, mid/mid diagonal cells. No teachers were located in the low/low diagonal cell of our sampling matrix. As such, our two low appropriation teachers had low levels of high scoring practice, but mid levels of growth on the two focal practices.
The cases were drawn from two schools in order to reduce the documented variation in teaching practices associated with school contexts. We sampled schools based on two criteria: sample size and variability in participation length among teachers, so we could analyze how duration of participation was associated with differential uptake of the PLATO practices. Ultimately, we focused on two schools, North Bend and Reed Valley, that had more than five teachers participating in the professional development across both cohorts of the study. Specific features of these two schools are included later in this paper. We sampled one teacher at the high, mid and low levels of appropriation from each of the two schools. This cross-case analysis of qualitative data provided a window into additional factors that may have supported or hindered teachers uptake of the professional development.
Professional development and school site meeting fieldnotes, transcripts, and debrief notes. All fieldnotes, debrief notes and transcripts of professional development and school site meetings were analyzed systematically, with multiple passes through each document by two researchers. After the identification of cases, researchers thematically coded all references to the targeted practices of Classroom Discourse and Strategy Use and Instruction, teachers descriptions of changes in their practice, challenges and successes they experienced. In addition, the professional development providers had noted that lesson and unit planning was a key practice that seemed to differentiate among teachers in the study. Thus, we also coded for references to lesson planning. These codes were then refined and categorized into larger themes, and memos were repeatedly written and shared among researchers (Dyson & Genishi, 2005). The memos allowed researchers to organize the findings thematically, and to compare findings across cases. These data were then examined for confirming and disconfirming evidence (Miles & Huberman, 1994), refining memos and claims as a result.
Interviews. All interviews were transcribed and then analyzed thematically for references to the PLATO practices, PLATO professional development, and descriptions of teachers practice. Using multiple passes through these data by two or more researchers, we identified themes in teachers responses during interviews and in their interactions with colleagues that cut across cases. We then systematically examined the data for all references to PLATO practices, changes in teachers practice, and teachers reflections on their experiences with PLATO, particularly the aspects of the experience that they found challenging and rewarding. Memos were generated to document the findings that emerged, first comparing case study teachers at each level of appropriation and then across cases. At each stage of the process, attention was again paid to confirming and disconfirming evidence (Miles & Huberman, 1994).
CASE STUDY SCHOOLS AND TEACHERS
As noted, we selected case study teachers from two schools that had participated across both years of the study, had the highest numbers of participating teachers, and that also demonstrated adequate variation in appropriation across practices.
OVERVIEW OF CASE STUDY SCHOOLS
North Bend serves approximately 600 students in grades 6 through 8. The student body is about 40% Hispanic and 40% Asian, and approximately 25% of its students are designated English Language Learners (ELLs). More than half of students qualify for subsidized lunch. Less than 15% of students receive special education services, while 17% are designated as Gifted. During the years of the study, 45% of the schools population achieved proficient or advanced proficient status on the states ELA exam.
Reed Valley serves 800 students in grades 6 through 8. The student body is majority Asian, with 16% Hispanic students. More than 75% of students receive subsidized lunch, about half are English Language Learners, 15% receive Special Education services, and 20% are designated as Gifted. Sixty percent of students achieved proficient or advanced proficient status on the states ELA exam during the years of the study. English/Language Arts class sizes are generally small in both schools, with an average teacher to student ratio of 1 to 18.
OVERVIEW OF CASE STUDY TEACHERS
Table 4 provides demographic information about our six case study teachers. At each level of appropriation, one teacher had a single-subject credential in English Language Arts, while another had a multiple subject credential. The two teachers with the highest levels of appropriation were also the most experienced teachers, although there were also relatively experienced teachers (i.e., more than 5 years of experience) at both the middle and lower levels of appropriation.
Table 4. Overview of Case Study Teachers
Table 5 demonstrates the variation in teachers mean scores, the growth in PLATO scores from the baseline observation to the highest observation cycle, as well as the percent of segments that scored at higher levels, or at the 3 or 4 score points in a given PLATO scale. There is clearly variation even between the two teachers in a given level. For example, Mr. Norris, one of the high appropriation teachers improved by more than a PLATO score point on both focal practices, while Ms. Rogers, the other high appropriation teacher section grew much more in Strategy Use and Instruction than Classroom Discourse. Nearly half of one of the mid-appropriation teacher, Ms. Reagan's, scores on Classroom Discourse were at the higher scores of 3 or 4, while she never reached the same bar for Strategy Use and Instruction. The other middle level appropriation teacher, Ms. Newell, was more consistent in her growth across the two practices, as well as in the frequency of higher scoring instruction. In the section that follows, we describe each case study teacher briefly.
Table 5. Case Study Teachers Performance on PLATO Practices
High Levels of Appropriation: Maureen Rogers and Damian Norris
Maureen Rogers had been teaching middle school for 24 years when she began the PLATO professional development. With the exception of 7 years in which she served as a Special Education teacher, she had taught 6th grade humanities for her entire career and was the most experienced teacher in the English department in Reed Valley. Her principal volunteered her to join the professional development as a member of Cohort 1. She was vocal about her hesitation around aspects of the professional development, including being audio and video recorded and attending sessions during summer break. When the PLATO professional development began, Ms. Rogers described entering the session expecting to have professional development focused on raising the kids test scores and this is how were going to do it, but was pleasantly surprised to find that it was really about delv(ing) deeper into the subject [and] mak[ing] sure the kids understand . . . and think. Despite her original hesitations, she described PLATOs vision for strong practice as lin[ing] up with [her] teaching philosophy.
Damian Norris, part of Cohort 1, was also a relatively experienced teacher. He had only been teaching at humanities at North Bend for a single year prior to the PD, but he had an additional 6 years of experience teaching high school English. During the first year of the project, he served as the chair of the North Bend English Department and successfully prepared his application for National Board Certification.6 Like Ms. Rogers, he began the project with palpable skepticism and was particularly wary of the observation component. However, by the end of the project, he described the professional development as being well-planned and a good use of time.
Mid Levels of Appropriation: Danielle Newell and Brittany Reagan
Danielle Newell taught 7th and 8th grade humanities in North Bend. She had 7 years of teaching experience at the end of the study and at one point had also served as the department chair. Before teaching at North Bend, she had received a multiple subject credential and been an English Language Development teacher. Ms. Newell was also part of Cohort 1, and she described herself as eager to join PLATO because she needed more engaging curricular materials and instructional techniques to better meet her students needs.
Brittany Reagan was a first-year teacher at Reed Valley who joined the PLATO professional development in Cohort 2 after graduating from a local masters program with a single-subject English and Social Studies credential. Ms. Reagan began attending the PLATO summer professional development before the academic school year began, and not surprisingly, had not attended any prior professional development. She described being excited about the professional development because it would give her a chance to collaborate with her new colleagues at Reed Valley.
Lower Levels of Appropriation: Alice Romney and Beatrice Nevins
Alice Romney described herself as a new teacher having been hired mid-year by Reed Valley during the 20102011 school year, just one year prior to participating in the PLATO professional development. She joined the PLATO professional development in summer of the second year in Cohort 2 and held a single-subject English and Social Studies credential. Throughout her interviews, Ms. Romney seemed unsure of her ELA-teaching skills. She mentioned being a novice teacher multiple times saying, I did my student teaching in social studies, not English. I relied pretty heavily on the book because Ive never taught middle school, and so I was really coming in without a lot of background to draw on. Given her short teaching career, it is not surprising that Ms. Romney did not speak at length about prior professional development experiences. Her one critique of prior professional development focused on their lack of immediate applicability to her classroom, saying that she left professional developments feeling like Okay, this is a great idea but I still dont really know how to do it in my class. She welcomed the chance to participate in PLATO and saw it as something to start with and some kind of protocol for, Heres how you teach English.
Beatrice Nevins was a more experienced teacher, who had been teaching middle school humanities at North Bend for 6 years and held a multiple subject credential. She joined the PLATO professional development project in Cohort 2. Despite her experience, Ms. Nevins, described feeling unprepared for the rigors of the classroom. Ms. Nevins was initially reluctant to participate in PLATO because she felt that historically the district changed instructional strategies so much from year to year that she could not buy into new approaches presented in professional development. Ultimately, however, she ended up appreciating PLATO because she perceived it as well aligned with what was already happening in her classroom.
This 2-year study includes a small sample of teachers involved in a professional development project at a specific school district that does not represent teachers or districts more broadly. Moreover, our qualitative case-study analyses are drawn from an even smaller sample of six middle school ELA teachers. Though the sample size is inherently limited, focusing on a smaller group of teachers allowed for more detailed data collection and analysis than a larger sample would have allowed. These rich, multifaceted data allow us to begin generating hypotheses about factors impeding or supporting the appropriation of ambitious instructional practice that could then be tested in larger and more representative samples of teachers.
We also recognize that we test a number of different hypotheses with our various statistical models. Were we making causal or even definitive explanatory claims, we would want to adjust our estimates for multiple hypothesis testing using a Bonferroni adjustment, especially given our small sample. That said, we see these analyses as exploratory, designed to surface potential factors impeding or promoting growth in teaching practices during and after the PLATO professional development. We hope that further research will also explore these factors using larger, more sufficiently powered samples.
In addition, while PLATO seeks to capture multiple elements of a teachers practice, we recognize that PLATO, like any standardized observation tool, is an incomplete assessment of a teachers instruction. There are numerous other indicators of instructional quality, including the strength of teacher-student relationships (Pianta, Mashburn, Downer, Hamre, & Justice, 2008) and the sophistication of the texts made available to students (Hoffman, Sailors, Duffy, & Beretvas, 2004). However, we can only speak to the specific teaching practices highlighted in the PLATO observation protocol.
The professional development may also have shifted teachers practice in ways not captured by the PLATO tool, such as the types of assignments and assessments teachers create or the feedback teachers provide on students written work. Self-reported interview data were triangulated with changes in teachers practice as measured by PLATO, as well as researchers observations during the professional development, observations and school site meetings. However, interviews only allow for teachers perceived shifts in practices, and are thus limited.
We also cannot make broad claims about the teaching practices of these teachers over the course of the entire school year, although we did purposively observe them at multiple time points using data from generalizability studies of PLATO (Cor, 2011). Teachers may well employ different practices when teaching different specific types of lessons, based on their knowledge and enthusiasm about a given topic. Though we found that the broad and readily measurable aspects of a lesson contentwriting, reading, grammar, etc.were not associated with instructional quality, there may well be more nuanced features of a lesson that impact the type and quality of practices teachers utilize.
Finally, our data do not allow us to make definitive claims about the reasons why teachers use the practices they do, or to make specific assertions about the ways in which schools or the composition of students in a particular classroom mediate the appropriation of the PLATO practices. This is, however, a first step, a descriptive analysis to explore the ways in which different factors are associated with higher and lower levels of uptake of ambitious teaching practices.
In the following section we detail four findings from the study that cut across both the quantitative and qualitative data. First, we found that the duration of professional development seemed to matter in terms of teachers appropriation of PLATO practices. Teachers who participated in a single year of the PLATO professional development appropriated the targeted practices less frequently and with lower levels of sophistication in practice use. The second key finding was that there were a set of what we term foundational practices that seemed to support the appropriation of more ambitious practices targeted in the PLATO professional development. These practices include Time Management and Behavior Management, two scales included in the PLATO tool, as well as instructional planning, a practice currently not featured in the observation system. Third, we found that teachers who appropriated the PLATO practices at higher levels understood the practices as conceptual tools that could and should be used in flexible and adaptive ways across lessons, rather than simply procedural ones that should be used in the same way modeled in the professional development. Finally, our findings reinforce a body of literature suggesting the value of stable and supportive communities for professional learning and growth (cf. Franke & Kazemi, 2001; Lieberman & Wood, 2002). Teachers who used their colleagues as resources for ongoing reflection and development appropriated the PLATO practices at higher levels.
TEACHER BACKGROUND CHARACTERISTICS AS PREDICTORS OF GROWTH
Analysis of teacher characteristics that predicted growth and sophistication of practice revealed that surprisingly few of the 16 teacher-level variables had a statistically significant relationship with either growth or sophistication of practice (see Table 6). For example, prior teaching experience in a specific school level (middle school, high school, or elementary school) or teaching in subjects outside of ELA (primarily science and social studies) did not significantly predict growth or sophistication of practice in the targeted elements. Years of service at the district or experience with leadership roles such as department chair or Beginning Teacher Support and Assessment (BTSA) mentoring were likewise unrelated to our outcomes of interest.
While only moderately predictive, the strongest predictors were related to training for and teaching experience with English Language Learners, specifically a Crosscultural Language and Academic Development (CLAD) credential and prior teaching experience in English as a Second Language (β = .083, p < .05); however bilingual CLAD certification was not predictive (see Table 7).7 Additionally, having a single-subject credential predicted growth in Classroom Discourse (β = .421, p < .01). Prior teaching experience in English Language Arts (β = .048, p < .05) and English as a Second Language (β = .182, p < .01) predicted growth in Strategy Use and Instruction. This could potentially be due to evidence suggesting that scaffolding classroom instructionfor example, being explicit about language goals, providing explicit instruction for features of Englishand classroom discourse are research-based practices for ELL students (Goldenberg, 2008). Thus, professional development and certificate training likely also privilege those practices, which have some overlap with both Strategy Use and Instruction and Classroom Discourse. Likewise, experience in teaching ELA and a single-subject credential may deepen a teachers understanding of the content area and allow them to more easily enact the targeted practices.
Note: each coefficient comes from a separate regression
* p < .05. ** p < .01
The findings from observations in teachers classrooms suggest that instructional practices generally increased with increased participation in the study. Table 8 demonstrates the difference in growth on the targeted instructional practices between Cohorts 1 and 2.
* p < .05. ** p < .01
Participants in Cohort 1, who received 2 years of professional development, had statistically increased growth in Strategy Use & Instruction (M = .69, SD = .58) as compared to Cohort 2, who received 1 year of professional development (M =.14, SD = .18); p=.007. While those in Cohort 1 also grew more in Classroom Discourse (M = 0.8, SD = .15; M = .55, SD = .09, respectively), the difference was not statistically significant (p = .18). While these differences in growth could be due to training foci differences between years 1 and 2 or selection bias into the cohorts, tests of differences in baseline PLATO scores were statistically indistinguishable from zero between Cohorts 1 and 2. This suggests the two cohorts of teachers were similar in their practice upon starting the PLATO professional development.
The qualitative data supported the quantitative finding that longer participation in the PLATO professional development promoted teachers uptake of practices. The three case study teachers who participated in PLATO for 2 years (two high appropriation and one mid appropriation teacher) seemed to benefit from longer participation in PLATO, as evidenced by the growth of their observation scores. For example, Ms. Rogers made continual growth in Strategy Use and Instruction in particular, beginning the study at a mean of 1.6 and continuing growth into the year after the study had ended with a mean score of 2.5. In contrast, the teachers at the low levels of appropriation described the challenges they faced in implementing the practices after one year of PLATO professional development, suggesting that more time in the professional development program would have enhanced their understanding of the focal practices and helped in incorporating them in their classrooms. For example, Ms. Nevins and Ms. Romney both attributed this lack of understanding about SUI to only participating in the second year of the professional development program. Ms. Nevins noted that she was a little sad that [she] didn't get to do the full two years because she would have understood the PLATO practices in a little bit more depth and felt more comfortable implementing them.
FOUNDATIONAL PRACTICES SUPPORT TEACHERS APPROPRIATION OF MORE AMBITIOUS PRACTICES
Findings from both the quantitative and case study analyses also suggest that classroom management techniques, what we term foundational practices for teaching, are predictive of growth and higher scoring practice. Time Management, in particular, was predictive of growth in Strategy Use and Instruction and higher scoring practice in both elements. A teacher with a one point higher average score on Time Management was associated with an average of .306 points of growth on Strategy Use and Instruction. That is, on average, teachers with higher Time Management scores were more likely to make growth in the element of Strategy Use and Instruction
Qualitative analysis of the six case study teachers also supported the quantitative data and previous literature, illustrating the importance of Time and Behavior Management as foundational practices. The teachers at high levels of appropriation on Strategy Use and Instruction and Classroom Discourse, the focal practices for the professional development, had Behavior and Time Management scores well above the mean. In contrast, the teachers at low levels of appropriation had Time and Behavior Management scores almost a full standard deviation below the mean. The teachers at the mid levels of appropriation were divided. One teacher, Ms. Reagan, scoring above the mean in Time and Behavior Management, and the other teacher, Ms. Newell, had Behavior Management scores below the mean. Notes written after each observation frequently described issues with Time and Behavior Management in many of the mid and low-appropriation teachers classrooms, noting Behavior Management as a pervasive problem and the teachers consequences as unclear and inconsistent.
This inconsistency in rules and consequences was often coupled with teachers overly ambitious expectations for the amount of work that students would be able to complete in a given time. For example, after an observation in Ms. Newells room, the researcher noted that the teacher had tried to accomplish A LOTdeveloping a character using a fill-in-the-blank worksheet, explaining how to show, rather than tell, what a character is like etc.while continually battling for her students attention. This type of description was common, distinguishing teachers Time and Behavior Management in the high appropriation classrooms from the management in the mid and low-levels of appropriation teachers classrooms.
In addition to stronger Time and Behavior Management skills, the case study findings showed that the high-appropriation teachers also demonstrated a deeper understanding of instructional planning in ELA than the other case study teachers. Both teachers at the high levels of appropriation were adept at sequencing lessons and decomposing complex ELA practices into smaller, sequenced lessons. They also had clear visions for how the skills within a lesson fit into broader, more long-term goals for developing students reading and writing skills. Fieldnotes from the professional development sessions noted that while other teachers were still getting settled into the task of planning a lesson using the PLATO practices, Mr. Norris was drafting a whole instructional sequence and using the model from last Friday as a template. Similarly, the professional developers described at several points in the debrief notes that they were not worried about Ms. Rogers planning skills, unlike many other teachers in the sample.
In the classroom observations and in the school site meetings in which they shared their practice with colleagues, these high appropriation teachers shared materials or made suggestions to colleagues that demonstrated their ability to couple materials from the PLATO professional development with classroom curricula to create multiday unit plans. For example, Ms. Rogers demonstrated evidence of careful and skilled planning in the instructional sequences she described during collegial consultations within the professional development sessions, as well as at school-site meetings. She often made suggestions to colleagues about the lessons they were in the process of planning and how they might fit into larger instructional sequences. Similarly, Mr. Norris colleagues at North Bend often described him as a resource for their own lesson and unit planning.
In contrast, the mid and low-appropriation teachers consistently struggled to envision big picture instructional goals and to then construct and sequence lessons to reach those goals. In interviews, these teachers themselves described that the sequencing of lessons and breaking the task down into smaller parts were obstacles to their implementation of the practices, particularly in relation to strategy instruction. Ms. Newell, a teacher at the middle level of appropriation, found it challenging to work backwards in planning a lesson from a broad instructional goal. Throughout the study, she described struggling to identify specific, discrete instructional objectives. She characterized her own tendency as a teacher as attempting to teach ten things all in one class. At school site meetings, this was evident in the numerous ideas she had for implementation of materials, without a clear connection to goals for student learning or a progression of lessons that might systematically teach students to meet her broader goals. She also described rushing through projects, not having time to read the essays students wrote, or being hurried to finish a novel. In one school site meeting she summed it up by saying, my biggest thing is just like not allocating enough time towards what Im doing. I just need more time. Professional developers also noted at numerous points that Ms. Newell found it very challenging to both envision a larger end goal, as well as how to break the instructional goal down into a sequence of lessons. For example, after one afternoon in the second year of the study, a professional developer noted, I spent almost the whole time with (Ms. Newell) and by the end I think she sort of had a handle on what she was going to teach and then added, but it is around this very piece (of knowing what to teach) that shes struggling.
In contrast, Ms. Reagans instructional planning practices were more illustrative of a new and inexperienced teacher. While Ms. Reagan seemed to be able to translate her larger goals for learning into individual lessons, she was still developing her capacity to sequence multiple lessons or create well-connected units. Observations of her teaching, as well as the lessons she shared at school site meetings, indicated that she was focusing on particular concepts or skills in an isolated rather than integrated way, switching from one focus to another throughout the course of the year without identifying connections between them. Each lesson was presented to her colleagues as a distinct entity, rather than a piece of a broader instructional puzzle. Throughout her year of participation in the study, observers noted that Ms. Reagan seemed to struggle with coherence in her planning.
The teachers at the lower levels of appropriation of practice found instructional planning to be particularly challenging. Ms. Nevins described that her teacher preparation program had not provided her with a foundation in curricular planning, and she felt unprepared for the rigors of the classroom. Ms. Romney described herself as someone who had big picture vision and in interviews articulated her large goals for students, but seemed to struggle to determine the steps needed to implement those goals. For example, in the final interview she described, I know what I want them to do, but Im not sure how to present it. Thats where I always get stuck. We want to write an essay. We want it to be great. How do we break that down to go from zero to great essay? What are the steps in between? Fieldnotes and debrief transcripts from the professional development also supported this, noting that planning an instructional sequence based on this larger goal was particularly challenging for both Ms. Nevins and Ms. Romney throughout their participation in the study.
UNDERSTANDING PLATO PRACTICES AS CONCEPTUAL OR PROCEDURAL TOOLS
Qualitative analysis of the case-study data demonstrated that teachers variation in the appropriation of PLATO practices was seemingly undergirded by the degree to which they recognized PLATO practices as conceptual versus practical tools. Both high-appropriation teachers began by using the materials as procedural tools, directly using texts or lesson plans taken from the professional development sessions but, over time, demonstrated a conceptual understanding of these tools (Grossman et. al, 1999). For example, Ms. Rogers and Mr. Norris both used multiple texts that had been introduced in PLATO professional development sessions. There were occasions where Mr. Norris implemented lessons that were directly part of the PLATO professional development, including a fishbowl lesson in which a small group of students modeled how to engage in discussion while the rest of the students watched. Similarly, in one school-site meeting, Ms. Rogers shared a week-long unit built around the short story Eleven, which had been used in a PLATO professional development. The unit incorporated multiple other lessons taken directly from the PLATO professional development such as opportunities for classroom discourse and explicit strategy instruction around how to identify character traits.
However, both teachers at the higher levels of appropriation went beyond using PLATO tools procedurally, modifying the resources provided in the professional development to meet their students needs and their instructional goals. Over time, they developed the sense that the practices were flexible tools that they could use in a variety of ways in their daily instruction. Ms. Rogers described a shift in her understanding of Strategy Use and Instruction like the light bulb going on, realizing that she could teach [students] how to do that exact thing that theyre not doing. Similarly, Mr. Norris described that PLATO had helped him to understand the value of classroom discourse and allowing students to speak in meaningful ways, a practice he used regularly across his lessons by the end of the project.
This conceptual understanding was apparent for both teachers across both focal practices. For example, after initially using the exact same strategies presented in the professional development, Mr. Norris regularly created his own strategy sheets for students for strategies such as summarizing or annotating a text. Similarly, while Ms. Rogers lessons were initially built upon artifacts or concrete tools from the PLATO professional development, she made alterations including adding visual supports through color coding and adapting the sentence frames suggested for discussion to support ELL students, expanding the original lesson to span across multiple days of instruction. Throughout the observations, interviews and school site meetings, it was clear that Mr. Norris and Ms. Rogers were comfortable modifying the PLATO practices and, by the end of the professional development, demonstrated a conceptual understanding of the PLATO principles that they used flexibly in their teaching.
This flexible view of PLATO aligns with Franke and Kazemis (2001) description of a generative view of learning from professional development, in which teachers continue to add to their understanding and make connections to create richly integrated knowledge structures (p. 105) driven by their own goals and inquiry. For both Mr. Norris and Ms. Rogers, viewing PLATO practices as conceptual tools allowed them to incorporate the PLATO practices generatively into their teaching. The practices were described as living, evolving things that were modified based on students needs, much the way Dreyfus (2004) describes that highly-skilled practitioners make subtle and refined discriminations as they apply strategies to particular situations.
The teachers at the mid levels of appropriation described PLATO practices in ways that demonstrated a developing, rather than consistent or complete, understanding of the practices. For example, while Ms. Newell described that Classroom Discourse was a familiar focal practice that had always been an integral part of her classroom experience, she wasnt able to describe the practice itself or her use of Classroom Discourse in the classroom in detail. Ms. Newell identified Strategy Use and Instruction as her largest area of learning in the project, but also indicated that she was aware that she had not mastered the practice and saw it as an area for continued growth. Similarly, while Ms. Reagan spoke of incorporating Classroom Discourse frequently, and feeling as though the practice felt natural and kind of like an extension of her work in her masters program, she described using Strategy Use and Instruction in a much more limited way, implementing it as a practical tool two to three times a unit for very concentrated topic[s]. Observations suggested that Ms. Reagan and Ms. Newell generally implemented PLATO practices as practical tools, rather than using them flexibly in ways that signaled a conceptual understanding of the practices. For example, Ms. Newell described that she observed a professional developer teaching a Strategy Instruction lesson around how to use dialogue tags, and responded with, This is an amazing lesson . . . Im going to go and do dialogue tags. However, there was little evidence of adaptation of the focal practices beyond the texts and activities presented in the professional development. Ms. Reagan also used the practices less consistently than high appropriation teachers. At times she incorporated a tool from the professional development procedurally (such as how to use sentence frames to spur discussion), but observations demonstrated that she was not consistently using the practice beyond that lesson. Thus, there was evidence that both mid level teachers demonstrated developing understandings of the focal practices, at times envisioning them as conceptual tools, but not consistently implementing them in ways that went beyond a procedural appropriation of practice in their classrooms.
In contrast, Ms. Nevins and Ms. Romney, the two teachers at the lower levels of appropriation, demonstrated emergent understandings of the practices, seeing them solely as practical tools and artifacts. They described implementing the practices as static and isolated activities, and neither teacher ever described these practices as conceptual tools they could use to plan instruction more generatively. For example, Ms. Nevins described PLATO as a set of materials: I mean I have a planner, a binder, of resources and then also the wiki site. Ms. Romney referred to the model lessons as very specific tool[s] [to] use. . . . And that was great for me because thats the part that I struggle with. She talked about PLATO as providing her with doable lessons that she could immediately implement in her classroom, noting that the professional development sessions provided materials she could take into [her] classroom and teach tomorrow. And thats what really helped me. Neither of these teachers adapted these resources to meet their students needs, nor incorporated the practices into other lessons regularly. They did not describe their use of the practices in ways that signaled they were beginning to conceptualize their instruction in terms of the practices more broadly.
COLLEAGUES SUPPORTED SOME TEACHERS APPROPRIATION OF PRACTICE
For teachers who had a developing understanding of the focal practices, colleagues played a key role in supporting teachers appropriation of the practices as conceptual tools. This appeared to be particularly true for teachers at the mid levels of appropriation. For example, Ms. Reagan described her school-site colleagues who attended the professional development as key resources in her own incorporation of practices and her ability to plan her instruction. Ms. Reagan described having most of the Reed Valleys department participating in PLATO as hugely beneficial. She described her department at Reed Valley as really willing to collaborate and come together, while also giving her the freedom to do [her] own thing in the classroom. In the final interview, Ms. Reagan described the collaboration as essential to the incorporation of PLATO in her classroom beyond the texts and activities modeled in the PD: being able to have people at the school site and then that time built into PLATO for collaboration made it that much more relevant. That it was like really stuff that we could apply and share.
There was also evidence that, even if these teachers themselves were not able to individually sequence instruction and integrate the PLATO practices into their instruction in generative ways, colleagues abilities to do so allowed teachers at the mid levels of appropriation access to the PLATO practices. For example, when Ms. Newell did feel like she had successfully incorporated strategy instruction into her teaching, she credited Mr. Norris for the lessons and materials saying:
I was so lucky because Mr. Norris had put together kind of like a strategy sheet, which is like this thing, and it made teaching that a piece of cake. Where I was like really starting to get nervous as I moved up to that point with the kids. Like, Oh god, I have to teach commentary. Its so difficult. But he made these sheets that just made it to easy. So they had every single tool.
Particularly for these teachers at the middle levels of appropriation, a supportive professional community appeared to play a key role in their ability to integrate PLATO practices more flexibly in their own instruction.
Teachers at the lower levels of appropriation did not appear to benefit from colleagues participation in PLATO to the same degree. While Ms. Romney, like other teachers at Reed Valley, was generally positive about her colleagues and described the department culture as steadily improving, she did not appear to use her planning time to collaboratively plan with any other teachers. At no point in the study, did she mention her colleagues practices as influencing her own. Ms. Nevins was less positive about the culture at North Bend, describing feeling unsupported and disconnected from the district and her colleagues in her department, noting that she was not a part of the department. She felt the sole exception to this was Mr. Norris, whom she sometimes bounced ideas off. However, she did not mention Mr. Norris nor her other colleagues as resources in her own implementation of PLATO. It appears that teachers colleagues within a collaborative teacher community may support some teachers ongoing appropriation of practice (Grossman, Wineburg & Woolworth, 2001). However, the lower appropriation teachers limited use of colleagues as resources also points to a more complex relationship between teacher learning and the professional teacher community and raises questions for further research about how such professional relationships could be leveraged to help all teachers develop conceptual understandings of practice.
Our quantitative and qualitative data suggest that the duration of participation in the PLATO professional development was associated with growth in the focal practices. Yet, as previous work has suggested (e.g., Franke & Kazemi, 2001), teachers uptake of the targeted practices varied considerably. The teachers at the lower levels of appropriation described that more time in the professional development program would have enhanced their understanding of the focal practices and helped in incorporating them in their classrooms. The need for time to develop ones teaching makes sense given our knowledge of teachers growth. Changing practice is indeed slow, steady work, and these practices were selected because they were challenging for teachers.
This study also suggests that there may be foundational practices that either support or hinder teachers uptake of the professional development practices. In this study, both time and behavior management seemed to be foundational practices that may have supported (or hindered) teachers uptake of PLATO professional development. However, this study also suggests that teachers abilities to plan instruction, decompose larger tasks into smaller ones, and situate current instruction in relation to larger ELA goals, might be vital to supporting teachers understanding of practices. Conversely, teachers who struggle to plan and sequence instruction may be hindered from using these practices flexibly as conceptual tools.
A key critique of professional development is the isolated nature of the learning experiences and the absence of transfer to teachers ongoing practice (cf. Yoon et al., 2007). This study suggests that strong foundational practices such as the ability to plan and sequence instruction may be crucial in supporting teachers generative use of practices targeted in professional development. While previous work (e.g., Penuel et al., 2007) has pointed to the importance of providing time to plan within professional development sessions, our study suggests that, in order to make the most of professional development, time to plan instruction is not sufficient. Teachers may require additional support to develop planning skills often associated with preservice or novice teacher support programs (John, 2006; Mutton, Hagger, & Burn, 2011). Like the teachers at the low and mid levels of appropriation in this study, even experienced teachers might need additional, concurrent support to build that foundation while targeting more ambitious, complex practices. Professional developers and school-based instructional leaders may need to consider how to provide differentiated opportunities in which teachers learn how to plan both lessons and units.
Finally, we see this study as highlighting the importance of assessing teachers planning abilities. While it is certainly difficult to observe teachers planning, our study suggests that it is a critical practice for new and experienced teachers alike, and thus a potentially important component of observation protocols. For example, the Framework for Teaching, an observation protocol used in many districts across the country, highlights the crucial importance of these skills, in the first domain, planning and preparation (Danielson, 2011). However, perhaps because the practices targeted in this domain are less readily visible and require an interview with a teacher, many districts do not in fact feature them in their evaluation systems (Kane, Taylor, Tyler, & Wooten, 2011; Taylor & Tyler, 2012). This study suggests that capturing planning practices and supporting teachers in engaging in high-quality planning is potentially quite important in supporting other efforts at instructional improvement.
Although collegial relationships between teachers were not the focus of this study, our findings suggest that for teachers at the midlevels of appropriation in particular, colleagues may serve as key sources of support in implementing this instruction in their classrooms, even when they are not yet able to conceptualize these practices generatively on their own. We see this as an area for further research, particularly longitudinal research, to explore how local collegial resources may help teachers improve over time, after the formal professional development has ended (Sun et al., 2013).
Furthermore, this study points to the need to differentiation within professional development. Just as we push teachers to tailor their instruction to individual students needs and meet them in the zone of proximal development, so too might we need to rethink the one size fits all model of teacher professional development. Effective professional development may not be effective for all teachers. Some teachers might need additional, concurrent support to build that foundation while targeting more ambitious, complex practices. The vital question is no longer whether professional development is effective, but rather for whom professional development is most effective and under what conditions. If professional developers are to move away from a one size fits all model of professional development and anticipate the heterogeneity in teacher development that this study and others document, then it will also be essential to explore how professional developers can anticipate and respond to teachers needs when designing and providing development. Teachers in this study did not receive individualized feedback about their teaching practice and classroom observations were used only for the purposes of diagnostic data collection. However, we now wonder if more individual coaching in conjunction with professional development would have provided opportunities for more tailored support. This represents a potential line of future research on how to best use standardized observation tools to leverage improvement in classrooms.
Finally, this study suggests that observation protocols can play a unique role in the design of professional development. In this case, PLATO allowed professional developers to gather standardized information across teachers, comparing changes in teacher practice in systematic ways. The use of a standardized assessment like an observation protocol might also be used in future work to help professional developers identify and respond to the heterogeneity in teachers practice that might affect uptake of professional development content. Observation protocols may make it easier to make comparisons about the effects of professional development across large groups of teachers and provide a means for measuring such change.
The PLATO tool made it possible to measure teachers Time and Behavior Management in this study. This suggests another benefit of using observation protocols in studies of professional development. Using multiple scales for multiple practices makes it possible to identify facets of teaching that shape teachers uptake of professional development, even when these practices are not the focus of the professional development itself. In addition to using observation protocols to explain and document ways in which teachers practice has changed, this study suggests that observation protocols may help professional developers to tailor instruction for teachers during professional development. Just as K12 teachers use pre-assessments of students knowledge and skills before beginning a new lesson or unit, so can observational tools provide diagnostic, baseline information that would allow professional developers to differentiate more effectively for teachers.
Many have argued that part of the challenge involved in improving the quality of teaching in our nations schools is the lack of valid and reliable measures for assessing teaching effectiveness or tools for targeting specific features of instruction (Gitomer, 2008; Grossman & McDonald, 2008). Without such tools, it is nearly impossible to identify effective classroom practices and support teachers growth in classroom instruction across the millions of teachers in American classrooms. As Lord Kelvin once said, it is impossible to improve at scale what one cannot measure. The recent developments in teacher evaluation have led to the development of such measurement tools. Measurement alone, however, will not lead to improvement unless we better understand the kind of sustained professional development that helps teachers improve their craft. With new tools at hand, research in professional development can take the next steps to push our understanding forward.
1. This study and all research methods were approved by Stanford Universitys IRB, as well as the participating districts IRB.
2. This teacher was included in the observation data for year one only.
3. A teacher who withdrew from the study mid-year was dropped from the analytic sample.
4. References to PLATO practices are capitalized throughout.
5. At the observational cycle level, Cronbachs alpha for the domains range from .66 (Instructional Scaffolding) to .81 (Classroom Environment).
6. The National Board Professional Teacher Certification process also promotes reflection on practice, which could have a relationship to his PLATO scores.
7. Because CLAD certification was required by the state at one point in time, we were concerned that it may have been proxying for years of experience. However, CLAD certification did not correlate with any prior teacher experience variable higher than .30. Likewise, certifications did not significantly predict baseline PLATO scores.
8. Some data for sample teachers in this table is from a survey administered by our research team. The data for district teachers come from district administrative data. We used the most complete data available for both groups; however, the sample teachers look more similar to district teachers than reported hereparticularly in White and Hispanic categorieswhen using administrative data only. This is due mostly to nonreporting of race data to the district.
Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333(6045), 10341037.
Bastian, K. C. (2013). Do teachers non-cognitive skills and traits predict effectiveness and evaluation ratings? Paper presented at Association for Education Finance and Policys 38th annual conference.
Beck, I. L., & McKeown, M. G. (2002). Questioning the author: Making sense of social studies. Educational Leadership, 60(3), 4447.
Borko, H., Jacobs, J., & Koellner, K. (2010). Contemporary approaches to teacher professional development. In P. Peterson, E. Baker, & B. McGaw (Eds.), International encyclopedia of education (Vol. 7, pp. 548556). Oxford, England: Elsevier.
Boyd, D. J., Grossman, P. L., Lankford, H., Loeb, S., & Wyckoff, J. (2009). Teacher preparation and student achievement. Educational Evaluation and Policy Analysis, 31(4), 416440.
Cor, K. (2011). The measurement properties of the PLATO rubric. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Danielson, C. (2011). Enhancing professional practice: A framework for teaching. Alexandria, VA: ASCD.
Desimone, L. M. (2009). Improving impact studies of teachers professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181199.
Dreyfus, S. E. (2004). The five-stage model of adult skill acquisition. Bulletin of Science, Technology & Society, 24(3), 177-181.
Duffy, G. G., Roehler, L. R., Meloth, M. S., Vavrus, L. G., Book, C., Putnam, J., & Wesselman, R. (1986). The relationship between explicit verbal explanations during reading skill instruction and student awareness and achievement: A study of reading teacher effects. Reading Research Quarterly, 21(3), 237252.
Dyson, A. H., & Genishi, C. (2005). On the case. New York, NY: Teachers College Press.
Franke, M. L., & Kazemi, E. (2001). Learning to teach mathematics: Focus on student thinking. Theory into Practice, 40(2), 102109.
Garet, M., Porter, A., Desimone, L., Birman, B., & Yoon, K. (2001). What makes professional development effective? Analysis of a national sample of teachers. American Educational Research Journal, 38(4), 915945.
Gitomer, D.H. (2008). Measurement issues and assessment for teaching quality. Thousand Oaks, CA: SAGE.
Goldenberg, C. (2008). Teaching English language learners: What the research doesand does notsay. American Educator, 10-43.
Good, T. L., & Grouws, D. A. (1977). Teaching effects: A process-product study in fourth grade mathematics classrooms. Journal of Teacher Education, 28(3), 4954.
Greenleaf, C., L., Schoenbach, R., Cziko, C., & Mueller, F. L. (2001). Apprenticing adolescent readers to academic literacy. Harvard Education Review, 71(1), 79129.
Grossman, P., Brown, L., Chambers Schuldt, L., & Cohen, J., (2014). How does an observation protocol inform an English Language Arts professional development on core practices? Symposium paper presented at the 2014 American Educational Research Association conference, Philadelphia, PA.
Grossman, P., Brown, L., Chambers Schuldt, L., Metz, M., & Johnson, E. M. (2013). From measurement to improvement: Leveraging an observation protocol for instructional improvement. Symposium paper presented at the 2013 American Educational Research Association Conference, San Francisco, CA.
Grossman, P., & McDonald, M. (2008). Back to the future: Directions for research in teaching and teacher education. American Educational Research Journal, 45(1), 184205.
Grossman, P., Wineburg, S., & Woolworth, S. (2001). Toward a theory of teacher community. The Teachers College Record, 103(6), 9421012.
Grossman, P. L. (1992). Why models matter: An alternate view on professional growth in teaching. Review of Educational Research, 62(2), 171179.
Grossman, P. L., Smagorinsky, P., & Valencia, S. (1999). Appropriating tools for teaching English: A theoretical framework for research on learning to teach. American Journal of Education, 108(1), 129.
Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 4551.
Henry, G.T., Bastian, K.C., & Smith, A.A (2012). Scholarships to recruit the best and brightest into teaching: Who is recruited, where do they teach, how effective are they, and how long do they stay? Educational Researcher, 41(3), 83-92.
Hill, H.C., Beisiegel, M., & Jacob, R. (2014). Professional development research: Consensus, crossroads, and challenges. Educational Researcher, 42(9), 476-487.
Hillocks, G. (1995). Teaching writing as reflective process. New York, NY: Teachers College Press.
Hoffman, J. V., Sailors, M., Duffy, G. R., & Beretvas, S. N. (2004). The effective elementary classroom literacy environment: Examining the validity of the TEX-IN3 observation system. Journal of Literacy Research, 36(3), 303–334.
John, P. D. (2006). Lesson planning and the student teacher: Re‐thinking the dominant model. Journal of Curriculum Studies, 38(4), 483–498.
Kagan, D. M. (1992). Professional growth among preservice and beginning teachers. Review of Educational Research, 62(2), 129169.
Kane, T. J., & Staiger, D.O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, Washington: Bill and Melinda Gates Foundation.
Kane, T. J., Taylor, E.S., Tyler, J. H. & Wooten, A. L. (2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46(3), 587613.
Kennedy, M. M. (1998). Form and substance in in-service teacher education (Research Monograph No. 13). Arlington, VA: National Science Foundation.
Knapp, N. F., & Peterson, P. L. (1995). Teachers' interpretations of CGI after four years: Meanings and practices. Journal for Research in Mathematics Education.
Little, J. W. (2003). Inside teacher community: Representations of classroom practice. Teachers College Record, 105(6), 913945.
Langer, J. A., & Applebee, A. N. (1986). Reading and writing instruction: Toward a theory of teaching and learning. Review of Research in Education, 13(1), 171194.
Lieberman, A., & Wood, D. R. (2002). The national writing project. Educational Leadership, 59(6), 4044.
Mashburn, A. J., Pianta, R. C., Hamre, B. K., Downer, J. T., Barbarin, O. A., Bryant, D., Burchinal, M., Early, D. M., & Howes, C. (2008). Measures of classroom quality in prekindergarten and childrens development of academic, language, and social skills. Child development, 79(3), 732749.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage.
Moschkovich, J. N. (2013). Issues regarding the concept of mathematical practices. In Y. Li & J.N. Moschkovich (Eds.), Proficiency and beliefs in learning and teaching mathematics (pp. 257275). Netherlands: Sense Publishers.
Mutton, T., Hagger, H., & Burn, K. (2011). Learning to plan, planning to learn: the developing expertise of beginning teachers. Teachers and Teaching, 17(4), 399416.
Nystrand, M. & Gamoran, A. (1991). Instructional discourse, student engagement, and literature achievement. Research in the Teaching of English, 25(3), 261290.
O'Connor, M. C., & Michaels, S. (1993). Aligning academic task and participation status through revoicing: Analysis of a classroom discourse strategy. Anthropology and Education Quarterly, 24(4), 318335.
Papay, J. P. (2012). Refocusing the debate: Assessing the purposes and tools of teacher evaluation. Harvard Educational Review, 82(1), 123141.
Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher, L. P. (2007). What makes professional development effective? Strategies that foster curriculum implementation. American Educational Research Journal, 44(4), 921958.
Pianta, R. C., Mashburn, A. J., Downer, J. T., Hamre, B. K., & Justice, L. M. (2008). Effects of web-mediated professional development resources on teacher-child interactions in pre-kindergarten classrooms. Early Childhood Research Quarterly, 23(4), 431451.
Snow, C. E., & Biancarosa, G. (2003). Adolescent literacy and the achievement gap: What do we know and where do we go from here? New York, NY: Carnegie Corporation.
Sperling, M., & Freedman, S. W. (2001). Research on writing. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 370389). Washington, DC: American Educational Research Association.
Sun, M., Penuel, W. R., Frank, K. A., Gallagher, H. A., & Youngs, P. (2013). Shaping professional development to promote the diffusion of instructional expertise among teachers. Educational Evaluation and Policy Analysis, 35(3), 344369.
Taylor, E. S., & Tyler, J. H. (2012). The effect of evaluation on teacher performance. American Economic Review, 102(7), 36283651.
Whitney, A. (2008). Teacher transformation in the national writing project. Research in the Teaching of English, 43(2), 144187.
Wilson, S. M., & Berne, J. (1999). Teacher learning and the acquisition of professional knowledge: An examination of research on contemporary professional development. Review of Research in Education, 24(1), 173209.
Yin, R. K. (2003). Case study research: Design and methods (3rd ed.). Thousand Oaks, CA: Sage.
Yoon, K. S., Duncan, T., Lee, S. W. Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the evidence on how teacher professional development affects student achievement. Issues & Answers. REL 2007-No. 033. Regional Educational Laboratory Southwest (NJ1).