Poor Implementation of Learner-Centered Practices: A Cautionary Tale

by Gina Schuyler Ikemoto, Jennifer L. Steele & John F. Pane - 2016

Many school systems are adopting new curricula in response to more rigorous standards that require higher-order thinking skills. This article presents implementation findings from a randomized, controlled trial of the Cognitive Tutor Geometry curriculum. We found a significant negative effect on student achievement despite the curriculum’s focus on learner-centered learning strategies that have previously been found to improve students’ ability to meet high mathematics standards. Our research confirms prior research that finds learner-centered instructional practices are correlated with higher student achievement. However, our findings also suggest that learner-centered curricula can actually do more harm than good when implemented poorly. We found that the cognitive demands of the curriculum coupled with teachers’ poor implementation of learner-centered instructional practices seemed to limit students’ ability to engage with the mathematical ideas. Teachers struggled to implement the curriculum because they lacked prior experience with learner-centered teaching strategies, had limited exposure to the curriculum, and were not provided with job-embedded support from principals or instructional leaders within their school. They also worked with students who were reluctant to collaborate and had low prior math achievement. Findings from this study suggest that curriculum adopters should be careful to ensure strong implementation of cognitively demanding curricula. In particular, districts and school leaders should provide intensive job-embedded professional development and support to assist teachers in achieving high implementation.

Many schools and districts are attempting to improve mathematics achievement and respond to higher math standards by adopting new curricula that embed research-based knowledge about how students learn and how particular types of instruction can support that learning (Remillard, 2005). Curriculum adopters, however, lack sufficient research knowledge about curriculum effectiveness to make informed decisions about which curricula are the most promising for their context (National Research Council, 2004; Schoenfeld, 2006). There is an urgent need for this information as schools and districts decide which curricula to adopt in response to the Common Core Standards.

The National Council of Teachers of Mathematics (NCTM) Research Committee (2008) and the National Research Council (NRC) report On Evaluating Curricular Effectiveness: Judging the Quality of K12 Mathematics Evaluations (NRC, 2004) have called for more rigorous evaluations of curricular effectiveness that include detailed implementation studies documenting not only whether particular programs work, but also the character of their implementation and the conditions under which they were implemented. When impact evaluations yield positive results, implementation studies can help program adopters understand and provide the conditions for effectiveness demanded by a particular curriculum (NRC, 2004; Ritter, Kulikowich, Lei, McGuire, & Morgan, 2007). Similarly, when impact evaluations yield negative results, findings from such studies can help program adopters decide whether to invest more to support implementation of a promising curriculumwith the hopes of improving its effectsor to abandon it for a more effective alternative.

This chapter responds to calls for more information about the character of implementation in context by reporting implementation results from a randomized, controlled trial of the Cognitive Tutor Geometry curriculum in Baltimore County Public Schools (BCPS). We decided to conduct a trial of this curriculum because it had shown promise in quasi-experimental studies, and a related curriculum, Cognitive Tutor Algebra I, was shown to have significant positive effects in a randomized field trial. We selected Baltimore County as the site for the study because it was a district needing to make substantial improvements in students mathematics achievement and wanted to adopt a new curriculum to meet that objective.  

However, our impact study found that the geometry curriculum had a significant negative effect on student achievement. Specifically, student achievement on the districts final exam was 0.19 standard deviation units lower in the treatment group than in the control group, which had been exposed to the districts standard geometry curriculum (Pane, McCaffrey, Slaughter, Steele, & Ikemoto, 2010). This finding raises several questions. First, to what extent did teachers implement the curriculum as intended? Second, which facets of curriculum implementation were most strongly linked to student achievement? Finally, what contextual factors were associated with the quality of implementation? This chapter addresses these questions and ends with recommendations for policymakers, curriculum developers, and researchers.

DESCRIPTION OF THE CURRICULUM INTERVENTION

The Cognitive Tutor series of curriculum products, which includes Cognitive Tutor Geometry, Cognitive Tutor Algebra I, and other courses published by Carnegie Learning, Inc., are complete courses designed to promote understanding of mathematics. The geometry course focuses on geometric concepts and principles and attempts to enhance abstract and spatial reasoning skills. The curricula include two components: teacher-guided classroom instruction (60% of instructional time), and individualized, computer-guided instruction using Carnegie Learnings tutorial software (the remaining 40% of instructional time). The classroom instruction and lab portions of the course are typically carried out in whole-class periods.   

The Cognitive Tutor softwarea central feature of these curriculawas designed by researchers at Carnegie Mellon University based on John Andersons ACT-R computational theory of thought (Anderson, 1983),  and draws on research from several disciplines, including artificial intelligence, cognitive psychology, and humancomputer interaction. The software contains a model of the processes students follow when solving problems, including multiple paths toward correct solutions and common misconceptions that lead to incorrect solutions. By tracing student progress in this model, the software intervenes with well-targeted feedback when the student makes errors, and continuously updates its assessment of skill mastery by considering the students correct actions, errors, and hint requests. The softwares adaptive nature is designed to provide individualized instruction to address students specific needs. Students are presented with challenging, multistep problems that reflect real-world situations and provide opportunities for students to build on prior knowledge and progress from concrete to abstract thinking.

In addition to the software, the curriculum provides student and teacher textbooks and workbooks, and detailed guidelines for instructors on structuring classroom time through learner-centered activities such as independent problem-solving, cooperative-learning group work, and student presentations. Typically, teachers receive three days of formal training by Carnegie Learning staff prior to curriculum implementation and an additional follow-up day after implementation begins. Teachers have access to an online professional learning community where they can access resources, share ideas, and seek feedback. Carnegie Learning also provides regionally based staff who visit schools to provide coaching assistance to teachers in their classrooms.

RESEARCH QUESTIONS AND ANALYTICAL FRAMEWORK

The research questions we address in this article are as follows: (1) What was the nature of implementation in treatment and control classrooms? (2) How is implementation related to student achievement? (3) What factors might explain the quality of implementation?

Our analytical approach to answering these questions responds to recent calls for curriculum-specific measures of textbook integrity (Brown, Pitvorec, Ditto, & Kelso, 2009; Chval, Chávez, Reys, & Tarr, 2008; Huntley, 2009) that recognize that high-quality implementation involves mutual adaptation in which teachers might appropriately alter the specifics of the written lesson to address specifics of their context while still honoring the underlying principles of the curriculum design (Ball & Cohen, 1996; McLaughlin & Mitra, 2001). At the same time, teachers may adopt curriculum materials without teaching in ways that align with the curriculum developers pedagogical orientation (Chávez, 2003; Chval et al., 2008; Tarr et al., 2008). Chval and colleagues (2008) outline

three essential components of textbook integrity: (a) regular use of the textbook by the teacher and students over the instructional period, (b) use of a significant portion of the textbook to determine content emphasis and instructional design over the school year, and (c) utilization of instructional strategies consistent with the pedagogical orientation of the textbook. (p. 72)

Because the Cognitive Tutor curriculum includes more than just textbooks, but also tutorial software and student workbooks, the concept of textbook integrity that we employ in this analysis includes measures of fidelity in use in all of the curriculum materials. We created an implementation rubric that would allow us to quantify fidelity of implementation along 18 different dimensions related to: scope and sequence, time in the computer lab, use of curriculum workbooks, use of curriculum homework materials, regular assessment, collaborative learning activities, making connections with prior knowledge and experiences, assignment of formal presentations, individualized learning, student explanations, student collaboration, student delivery of presentations, student engagement in classroom, student engagement in computer lab, teacher assistance, closing an activity, grading, and use of Cognitive Tutor software reports.

Our rubrics are similar to analytical tools that were developed to measure implementation of mathematics curricula, such as Math Trailblazers (Brown et al., 2009),  Connected Mathematics and MathThematics (Hord, Stiegelbauer, Hall, & George, 2006). On these analytical tools,

various dimensions of instruction are classified along a continuum from being very close to what the developer had in mind to a distant zone for which what is being done would be far from the developers intent. In this way, [the tools] yield continuous measures along several dimensions of instruction, rather than a dichotomous judgment of whether instruction has been faithful to the authors intents. (Huntley, 2009, p. 358)

The process we used to develop the implementation rubric began with two researchers attending the three-day Cognitive Tutor Geometry training for teachers and recording trainers descriptions of what constituted appropriate implementation, including messages about when and how teachers were expected to use their judgment to deviate from the planned curriculum. We drew on these notes and a thorough review of the curriculum materials to create a first draft of the implementation rubric, which we then provided to the developers for feedback. The rubric addressed mutual adaptation by clarifying what elements of the implementation guidance were expected to remain tight versus loose. We revised the rubric accordingly and then jointly observed three separate lessons with a Carnegie Learning representative. We discussed the observed lessons in relation to the rubric and added further clarification to the rubric in some places to improve reliability of the ratings. Research team members also attended the training provided to teachers participating in the study and confirmed that Carnegie Learning was giving teachers in our study the same messages about expectations for implementation.

RESEARCH METHODS

SETTING AND STUDY DESIGN

The study was carried out in eight high schools in BCPS, which is located in the urban fringe of Baltimore and serves students from a wide range of racial/ethnic and socioeconomic backgrounds. At the time of the study, the district had 25 high schools, which together enrolled about 32,000 students.¹ In the eight participating high schools, the average minority enrollment rate was 46%, and 26% of students qualified for free or reduced-price meals (FARMS).

Data collection was conducted during the 20052006, 20062007, and 20072008 academic years. The eight study schools arranged their schedules such that two participating teachers were scheduled to teach geometry concurrently during two class periods. During the first period, we randomly assigned one teacher to teach the Cognitive Tutor Geometry treatment group curriculum while the other taught the BCPS control group curriculum. In the later period, the teachers switched curricula. As such, the study involved two teachers and four classes per school in each year that a school participated.² Due to the departure of three teachers from their schools between years, a total of 19 teachers participated in the study. Students whose schedules called for them to take geometry during one of the two class periods under study were randomly assigned to either a treatment- or control-group classroom, regardless of their grade level or other characteristics. The randomized analytic sample consisted of 699 students, 351 of whom received the treatment curriculum, and 348 of whom received the control curriculum.³

DATA COLLECTION AND MEASURES

Classroom Instruction Measures

To monitor implementation of the treatment curriculum and measure instructional practices in both treatment and control classrooms, researchers visited each teacher in the study three times per year by appointment. During each visit, researchers conducted structured teacher interviews and observed each teachers treatment and control classrooms for an entire class period. At least one of the three treatment classroom observations was conducted during a class period during which students were engaging with the Cognitive Tutor software. The researchers recorded a running narrative of teaching and learning. Subsequent to the observation, researchers rated each class according to the implementation rubric and recorded justifications for each rating, which permitted subsequent data cleaning to ensure consistency across raters and years. Scores of one or two indicated that there was little to no evidence that practices related to that dimension were being implemented, while scores of three or four indicated some evidence, and scores of five or six indicated strong evidence.

To ensure consistency in classroom observations, all researchers participating in classroom observations attended Carnegie Learnings three-day training sessions for teachers, followed by three days of observation and interview training. During this training, the team jointly observed five different classrooms and compared individual ratings of these classrooms. For the last classroom observed, we calculated Cohens kappa, which ranged from 0.8 to 0.85 each year, suggesting strong agreement among raters.

To summarize these ratings, we averaged each teachers scores on each dimension for each academic year. We then averaged 16 of the 18 teaching practice dimensions into four unweighted composite implementation measures: (1) learner-centered practices, (2) student participation, (3) materials fidelity, and (4) computer lab use. In addition to constructing the unweighted composites, we used principal components analysis with varimax rotation to construct an alternative set of weighted composites, but the weighted principal components did not lend themselves to intuitive interpretations, so we used the unweighted composites in the analyses. Like the underlying dimensions, the values of each composite ranged from 1, signifying weak compliance with curriculum specifications, to 6, signifying strong compliance.

Student Achievement Measures

Student achievement in geometry was measured using the districts required geometry final examination, administered at the end of the course. The examination included 30 multiple-choice items and 11 extended-response items. Extended-response items were scored by experienced geometry teachers hired by the authors. To increase the precision of our estimates, our analysis also controlled for students prior achievement in mathematics using a 25-item, multiple choice, end-of-course algebra examination published by Educational Testing Service (2004). This pretest was selected because algebra precedes geometry in the districts curriculum. Pretest scores in the sample were low on average. Out of 25 possible points, the mean raw score was 8.3, with a range from 0 to 19. Both tests were standardized to a mean of 0 and a standard deviation of 1.

FINDINGS ON THE NATURE OF IMPLEMENTATION IN TREATMENT AND CONTROL CLASSROOMS

As mentioned above, treatment group students scored significantly lower on the post-test than control group students, and the difference of 0.19 of a standard deviation is educationally meaningful. Specifically, it is large enough to move a student from median performance to the 43rd percentile. The negative effect raises the question whether the curriculum was implemented as intended by the curriculum developers. Our analysis of implementation ratings found that:

"

Implementation of learner-centered practices was slightly higher in treatment classrooms, but relatively low overall.

"

Student participation levels were similar and modest in treatment and control classrooms.

"

Fidelity to Cognitive Tutor materials was reasonably high, ranging from medium to high.

"

Computer lab use was relatively consistent with curriculum guidelines.

The sections below elaborate on these findings.

IMPLEMENTATION OF LEARNER-CENTERED PRACTICES

Our first composite variable, learner-centered practices, captured the extent to which a teacher, during a given class period and school year: assigned presentations, encouraged students to make connections among concepts, brought closure to activities, and encouraged students to collaborate with one another. All are teaching strategies explicitly prescribed by the Cognitive Tutor curriculum, though they can also be measured in non-treatment classrooms. Developers expected teachers to use all of these. Figure 1 presents box plots showing the distribution of each component dimension in both treatment (top) and control (bottom) classrooms, on a 1 to 6 scale, with 1 indicating very low use of this strategy and 6 indicating very high use. The median level on three of the dimensions was higher in treatment classes, as one might expect, though the extent to which teachers encouraged students to make connections among ideas or concepts was virtually identical in treatment and control classrooms.

Figure 1. Learner-centered practice indicators (n = 30 treatment and 30 control classes)

Carnegie Learning suggests that teachers assign presentationsin which students are expected to share ideas, strategies, and knowledgein at least four out of every 10 class periods. An example of an intended presentation is included later in this paper, in Figure 7. However, we found low implementation of this practice across our treatment classrooms, with the majority of teachers assigning presentations two times or fewer per quarter. Nevertheless, this degree of implementation was still greater than in control classrooms, where most teachersexcept for a few outliersnever assigned presentations.

The Cognitive Tutor curriculum materials also emphasize making connections with prior knowledge and experiences. We found that the median implementation of this strategy was low, at 2 on a 6-point scale, in both treatment and control classrooms. A rating of 2 suggests that the teacher encouraged students to make connections one time or fewer each class period. The distribution of the practice had a wider upper range in treatment classrooms, suggesting that at least some teachers encouraged connections more often with the Cognitive Tutor curriculum than with the control curriculum.

Bringing closure to activitiesby asking summary wrap-up questions or reviewing student work in ways that ensured students understood the mathematics behind their solutionswas slightly more evident in treatment classrooms than in control classrooms, with a median of 2 in the former and approximately 1.6 in the latter. However, these medians still fell on the low end of the scale, meaning that teachers typically did not attempt to bring closure to activities or did not do so in ways that checked for students understanding. Students were often allowed to work until the bell rang, a point we return to in the discussion of Figures 8 and 9 below.

The largest difference we found in learner-centered practices between treatment and control classrooms was in the extent to which teachers encouraged student collaboration. The median rating for Cognitive Tutor classrooms was a 3, suggesting that the median teacher was assigning, monitoring, and supporting collaborative learning activities in approximately 30% to 80% of classroom lessons. This level of implementation did not meet Carnegie Learnings expectation that collaborative learning would be used during 80% of classroom lessons, but it was substantially higher than in the median control classroom, where collaborative learning was reportedly assigned during fewer than 30% of classroom lessons and was thus infrequently witnessed by our observers.

STUDENT PARTICIPATION

The next composite, student participation, was an unweighted average of three components that reflect students actions in the classroom, with a focus on behaviors that are encouraged by the Cognitive Tutor curriculum in both the computer lab setting and the regular classroom setting. The box plots in Figure 2 illustrate the distributions of these components in treatment and control classrooms. They suggest, notably, that median student engagement was higher in the control classrooms than in the treatment classrooms, but that engagement generally fell in the medium range, with interquartile ranges between about 3 and 5 in both treatment and control classes. This suggests that the majority of students were engaged the majority of the time in both types of classrooms. A medium rating also suggests that a handful of students were off task for a majority of the time. We also found that the median observed frequency of students explaining their thinking was almost as high in control classes as in treatment classes. However, the median observed frequency was still borderline low/medium in both cases, suggesting that, at best, there tended to be only some examples of students explaining their thinking and that these instances tended to be teacher-prompted. We did not observe many examples of students routinely explaining their thinking (by describing the process they used, solution strategies tried and discarded, or alternative solutions) to group members or the whole. When students were assigned to work in groups, we found that they tended to be distracted and did not show real signs of collaboration in either the treatment or control classrooms, although they were slightly more collaborative in treatment classrooms.

Figure 2. Student participation indicators (n = 30 treatment and 30 control classes)

IMPLEMENTATION OF COGNITIVE TUTOR MATERIALS

The next composite, materials fidelity, was based on six dimensions that were only observed in treatment classrooms because they were not applicable to control classrooms. The box plots in Figure 3 suggest that curriculum compliance was highwith medians between 5 and 6regarding regular use of the Cognitive Tutor workbooks and regular assignment of homework. Implementation on three of the dimensions tended to be at medium levels: Teachers were assessing students, but not as regularly as Carnegie Learning suggests; they were attempting to follow the prescribed pacing and ordering, but often fell behind on pacing. And they were using the software to inform instruction, but not in the more sophisticated ways that Carnegie Learning had demonstrated during training. Carnegie Learning suggests that teachers regularly access the Cognitive Tutor software reports and adjust or individualize their teaching accordingly, but teachers tended only to access the reports at the end of marking periods for the purposes of generating grades. The lowest area of compliance in Figure 3 was the extent to which teachers assigned grades that explicitly reflected student collaboration, computer lab use, and presentations. Carnegie Learning recommends that grades reflect these elements and that students be made aware of this grading practice. However, grading practices often did not place emphasis on student collaboration or presentations. In many cases this may be because the teachers were not implementing these practices as intended by Carnegie Learning. Even when the grades did reflect these elements, students were not necessarily made aware of this practice.

Figure 3. Materials fidelity indicators (n = 30 treatment classes)

COMPUTER LAB USE

The computer lab use composite was also based on dimensions only observed in treatment classrooms. The components of computer lab use are shown in Figure 4, where we observed that most teachers complied with the guideline that 40% of class time be devoted to students self-paced use of the Cognitive Tutor software. Median implementation of this practice was approximately 5.4 on the 6-point implementation rubric. Student engagement and teacher encouragement of computer lab work were also observed with moderately high frequency, with median implementation levels of 4 for each component.

Figure 4. Computer lab use indicators (n = 30 treatment classes)

CLASSROOM INSTRUCTION COMPOSITES

Figure 5 summarizes the distribution of the four instructional composites detailed above. These composites were used in the achievement analysis discussed in the following section. Having averaged the components of each composite, we found that fidelity of implementation of the curriculum materials and computer lab was moderately high. Implementation of learner-centered practices was fairly low in treatment classrooms, but slightly higher than implementation of the same practices in control classrooms. Student participation was moderately low in both the control and treatment classrooms.

Figure 5. Distributions of the Four Instructional Composite Variables (n = 30 treatment and 30 control classes)

RELATIONSHIP BETWEEN IMPLEMENTATION AND STUDENT ACHIEVEMENT

To address our second research question regarding how teachers implementation of the Cognitive Tutor curriculum is related to students achievement, we conducted two types of analyses: first, correlational analyses between curriculum implementation and students geometry achievement, and second, grounded theory analyses of classroom observation records.

CORRELATIONAL ANALYSES

Here, we summarize and reflect on results presented in detail in Pane et al. (2010). The results are correlational and therefore cannot be interpreted as causal. Moreover, because the composites were measured at the classroom level, with only 30 treatment and 30 control classes in the analysis, we had very limited statistical power to detect even correlational relationships between classroom practices and student achievementparticularly for the implementation composites measured only in the 30 treatment classes.

Bearing in mind these limitations, we found that two of the four instructional composites showed substantial positive relationships to students geometry achievement, holding constant individual and classroom-average pretest scores and classroom average FARMS eligibility. Specifically, an additional unit of learner-centered practices on the six-point scale was associated with an additional 0.43 standard deviation units of student achievement in geometry at the marginal 10% level of significance (p = 0.06). Importantly, this relationship did not differ significantly between treatment and control classes, meaning that the positive relationship between learner-centered practices appeared to be independent of the curriculum used. This relationship is illustrated in Figure 6, in which the estimated slope of the relationship between learner-centered practices and student achievement is the same regardless of whether these practices were implemented in treatment or control classrooms.

Figure 6. Fitted effect of learner-centered practices on student achievement in treatment and control classes (n = 699 students; 60 classrooms)

GROUNDED THEORY ANALYSIS

To further explore whether and how implementation influenced student achievement, we conducted a grounded theory analysis (Creswell, 1998; Glaser & Strauss, 1967) of narrative observation notes. Over the course of the study, observers wrote reflective memos and engaged in debriefing conversations to generate hypotheses about how implementation might be influencing student engagement and achievement. After preliminary student achievement analyses became available, we also generated hypotheses specifically with regard to how the curriculum might be generating a negative effect on student achievement. For example, our sense was that students tended to be more off task when asked to engage in groups (more often in treatment classrooms) than when they were asked to engage in teacher-directed instruction (more often in control classrooms).  We then revisited the narrative accounts of our observations to search for confirming and disconfirming evidence for our hypotheses.   

We found that the cognitive demands of the curriculum coupled with poor implementation of learning-centered instructional practices seemed to limit students ability to engage with the mathematical ideas. In Appendices A, B, and C, we present three vignettes that illustrate how the curriculum and instructional practices interacted to create different learning experiences for students in three classrooms. Appendix A provides an example from the high-fidelity classroom we visited as part of our implementation rubric development process. This school was in a different district; it had a similar percentage of students qualifying for FARMS (28% compared to 26% in the study sample) and a greater percentage of minority students (71% compared to 46% in the study sample). Despite a slightly different student population, this example is valuable because the curriculum developers selected it themselves as emblematic of the intended curriculum and it thus serves as a useful anchor for understanding the other vignettes.  

Appendix B provides an example of a typical treatment classroom, and Appendix C provides an example of the same teacher in a typical control classroom. Although the observers selected these vignettes, we confirmed that they are typical in that their implementation ratings generally match the median ratings for both types of classrooms. We chose to highlight implementation examples from the regular classroom setting rather than the computer lab setting because the curriculum called for students to spend more time (60%) in the classroom setting, and because it allows us to illustrate contrasts to the control classroom (which did not use the computer lab). However, we did find similar patterns of low implementation of learner-centered practice in narratives from the computer lab settings.

The example in Appendix A represents high fidelity in not only material use but also in implementation of learner-centered practices. For example, the teacher effectively assisted students in making connections between their own lives, the sailboat race problem, and the mathematical ideas related to triangles. She also had classroom norms and routines (such as desk configurations, standing group assignments, and expectations for group participation) that facilitated student collaboration. She also assessed student understanding and brought closure to the lesson by having the groups present their work. Throughout the lesson, the teacher facilitated students ability to engage in the mathematical ideas by scaffolding the prior knowledge and connections they needed to understand the problem and holding students accountable for supporting one anothers learning. This illustration of how the developers intended their curriculum to be implemented provides a useful contrast to explain how the example of a typical treatment classroom fell short of these intentions.

The examples from our study classrooms (in Appendices B and C) fit with the overall implementation findings that we described at the beginning of the section. That is, the teacher in the treatment classroom example typified the broader trend of using the Carnegie Learning materials and making more of an effort to support collaborative learning and other learner-centered practices in the treatment classroom than she did in the control classroom. However, she also typified the larger group in that she was unsuccessful in implementing these practices at the high levels intended by the curriculum developers. In both classes, she faced significant problems with student engagement.  

Further analyses of our observation narratives suggest that these classroom examples typify the broader sample in other ways as well. We found that when students were faced with the Cognitive Tutor Geometry curriculum, they were more likely to become stuck than students facing the control class curriculum. A comparison of the assignments suggest this should not be surprising given that the Cognitive Tutor assignments tend to pose complex, real-world mathematics problems while the assignments in control classrooms tend to ask straightforward procedural questions. Unlike in the high-fidelity classroom, teachers in the treatment classrooms did not tend to provide introductions to lessons that assisted students in making connections to prior knowledge and life experiences that might have helped them to make sense of the word problems. Also unlike the high-fidelity classroom, students in the treatment classrooms were unlikely to seek much support from other students when they became stuck. This may have occurred because they believed their peers lacked sufficient mathematical knowledge to help them and/or because their teacher did not implement strategies to hold students accountable for collaborating with one another. We explore possible reasons for the overall low implementation of the learner-centered practices in the following section.  

       

FACTORS THAT EXPLAIN QUALITY OF IMPLEMENTATION

In this section, we report findings regarding variables that enabled and hindered implementation of the Cognitive Tutor geometry curriculum. In particular, we provide explanations for why implementation of learner-centered practices tended to be low across our study classrooms. Similar to the previous analysis, we used a grounded theory approach to identify themes emerging from our data. This process led us to examine the following factors, many of which have also been identified by prior research: teacher buy-in, administrative support, type and intensity of professional development, technological resources and support, principal and department chair turnover, computer lab type (separate lab vs. in-classroom laptops), and the role of teachers and students prior experience with learner-centered instruction.

We used a grounded theory approach in the Glasserian tradition (Glaser, 1978), which makes use of all available dataincluding both qualitative and quantitative datato explore themes across our cases. We systematically coded teacher interviews (which included questions about factors that enabled or hindered implementation), classroom observations, training observations, and communications with principals and department chairs. For several of the factors, we created quantifiable codes and conducted exploratory correlational analyses to examine relationships between the factors and levels of implementation. We also created matrices (Miles & Huberman, 1994) to examine patterns in qualitative data.   

While it would be impossible to be certain that findings from a grounded theory analysis provide complete and accurate explanations of the case, our study design included several features that improve our confidence that our findings actually do fit the data. First, the randomized control trial design of the larger studywhich allowed us to collect data about the same teacher using two different curriculaprovided a useful framework for systematically sampling classroom observations and for exploring countervailing evidence (Miles & Huberman, 1994). Our study also benefitted from prolonged engagement in the field (Eisenhart & Howe, 1992)including interacting face-to-face with each teacher at least seven times per year. Repetitive observations, interviews, and conversations provided opportunities to collect missing data and confirm or improve the accuracy of previously collected data.  

We dismissed many of our original hypotheses (such as lack of teacher buy-in and competing messages from school leadership) due to lack of supporting evidence or the presence of contradictory evidence. However, we did find that, in general, there were certain conditions present across the study sample that seemed to enable implementation, and other conditions that seemed to hinder implementation. At one level, our study schools appeared to be a fertile place in which to conduct our experiment.

Strong Teacher Buy-In

We found that the majority of teachers were generally positiveand in some cases excitedabout the Cognitive Tutor curriculum.

No Competing Priorities

Although teachers reported that none of the principals or department chairs were closely monitoring implementation or providing teachers with opportunities for professional learning and collaboration to support their implementation, teachers did report that school administrators were generally encouraging teachers to implement the curriculum and were providing them with the resources to do so. We also found that teachers perceived their administrators to be supportive of the curriculum. Perhaps just as importantly, teachers did not report that they were faced with competing priorities and demands that may have hindered their implementation.

Resources

Almost all of the teachers reported having all of the resourcessuch as materials and technologythat they needed to implement the curriculum.

On the other hand, we found that classrooms in our study were a challenging setting for several reasons.

Lack of prior experience with learner-centered teaching. Teachers and students had limited experience with learner-centered teaching and learning. Only four teachers reported that they had attempted to implement learner-centered practices prior to the study. Our observations in control classrooms documented that almost all teachers in the study tended to employ more traditional question-and-answer practices. Not surprisingly, when we controlled for teaching experience and year in study, we found that teachers self-reported prior experience with learner-centered teaching practices was positively associated with their use of learner-centered practices in study classrooms. In particular, having experience with learner-centered practices prior to using the Cognitive Tutor curriculum predicted an additional 0.64 points on the six-point scale of implementation of learner-centered practices (p = 0.07). Though the sample is obviously too small for generalization, the lesson is that within these data, prior use of such practices was observably related to future use. We did not find statistically significant relationships between implementation levels and any of the other factors that we examined, including the teachers year in the study (among those who took part in the study for at least two years).

Students reluctant to collaborate. Teachers also reportedand we often observedthat students in the majority of classrooms were reluctant to collaborate with other students. During collaborative time, we frequently observed students off task (for example, socializing, texting, or sleeping). Teachers efforts to motivate or hold students accountable for engaging in the lesson often failed. Given that teachers were attempting to implement collaborative learning time more often in the treatment classrooms, they may have been inadvertently providing students with more opportunities to disengage. As such, the negative treatment effect was likely caused at least in part by poor implementation that resulted in low levels of engagement with mathematics for students in the treatment classroom.

Low prior math achievement. The study classrooms were also a challenging context in that students tended to be struggling math learners, as evidenced by their pretest scores and our classroom observations. This may have greater importance in the treatment curriculum because it requires students to solve word problems. When students lacked prior knowledge, they often became stuck and stopped working on the task while they waited for the teacher to become available to answer their question. As illustrated in the example of a high-fidelity classroom in Appendix A, Carnegie Learning intends for students to ask one another for assistance in these situations, but this strategy seemed particularly problematic in a setting where the majority of students lacked prior knowledge and a culture of collaboration. As such, we frequently observed teachers in treatment classrooms calling a halt to collaborative learning time in order to walk students through the problems in a direct instruction format. We also found that the typical treatment lesson moved at a much slower pace than curriculum developers intended, usually covering only half of the planned activities. Meanwhile, in control classrooms, we often witnessed a more deliberate scaffolding of the knowledge students needed to solve practice problems (such as provision of a set of practice problems that slowly increased in difficulty), and more problems being covered in a single period.

Limited exposure to the curriculum. A third challenge in the study was that most of the teachers (11 of 19) only taught the curriculum for one year. We designed the study with the intention of following teachers for multiple years so that we could understand whether and how quality of implementation changes over time. However, through a combination of teacher and school turnover within the study, our dataset only included data for eight teachers implementing the curriculum for a second year and, among those, three teachers implementing the curriculum for a third year. Therefore, the overall low levels of implementation might be explained by the fact that most teachers were new to the curriculum. Nevertheless, data from the teachers who implemented the treatment curriculum for a second and third year suggested that their implementation did not improve over time. In fact, the overall implementation scores of four teachers (of the eight total teachers who were in the study for at least two years) were slightly but non-significantly lower in their second year of implementation than in their first.

Lack of job-embedded development for teachers. A fourth challenge was that teachers did not participate in ongoing, job-embedded professional development to support their implementation. Most teachers did participate in three days of training provided by Carnegie Learning prior to the start of their first year using the curriculum. Carnegie Learning also provided a consultant who was available to visit teachers in their schools and deliver coaching and feedback. Teachers reported that while this person did check in on them once and answered some simple questions, the support did not substantially affect their instruction. On two unplanned occasions we had the opportunity to observe the consultants visit and interactions with the teacher. In both cases, the consultant asked the teachers if they had any questions at the end of the period as students transitioned in and out of the classroom, and the teachers responded that they did not have questions. This was not surprising to the observers given that there was not a time and space for teachers to initiate a conversation lasting more than a minute or two. None of the schools had school-based math coaches, teacher leaders, department chairs, or other school leaders that supported teachers general professional growth. Nor did teachers have common planning times and/or a school culture that facilitated teacher collaboration. As such, the teachers did not engage in the types of ongoing, job-embedded professional developmentincluding detailed feedback on observed practicethat characterizes high-quality professional development. Some teachers did attend more or less of the professional development that was offered for the curriculum, but this variation was not significantly correlated with implementation measures.

DISCUSSION

Many school systems are adopting new curricula in response to more rigorous standards that require higher-order thinking skills. Our research confirms prior research that finds that learner-centered instructional practices are correlated with higher student achievement. However, our findings also suggest that learning-centered curricula can actually do more harm than good when implemented poorly.  

Based on detailed observations, it seems that teachers struggled to implement the student-centered practices prescribed by the curriculum. Although they tried to implement collaborative learning by placing students into groups, this strategy left many students off task when they became stuck and waited for teacher assistance rather than collaborating with other students. While teachers had positive attitudes about the curriculum, they did not have the prior knowledge, curriculum experience, or ongoing, job-embedded professional support that might have helped them to implement the student-centered practices with students who lacked prior knowledge and willingness to collaborate with other students. These findings are not surprising given research documenting the need for detailed observation and feedback from instructional leaders (such as department chairs, coaches, or principals) to improve teacher practice. However, findings from this study raises grave concerns about the consequences of implementing curricula that require changes in teacher practice without adequate support, since students can be worse off when research-based instructional practices are implemented poorly.  

These findings also support prior research documenting challenges related to implementing learner-centered practices (see, e.g., Desimone, Smith, Baker, & Ueno, 2005). Our findings also extend this research by suggesting that curriculum materials by themselves cannot overcome these challenges. That is, curricula and software cannot make learning experiences for students teacher proof. This finding is particularly noteworthy given that some adoptersdespite cautions from the developerview the software component of the Cognitive Tutor curriculum as an easy way to provide students with individualized tutoring without having to train teachers to implement difficult instructional practices.

These implementation findings are important, particularly because they offer insights to policymakers trying to determine whether and how our studys overall finding of a negative treatment effect of the Cognitive Tutor geometry curriculum might apply to their situation. First, these findings suggest that policymakers in a similar contextwith students who lack prior math knowledge and willingness to collaborate, teachers who lack prior experience with learning-centered teaching practices, and schools that lack structures for providing teachers with job-embedded professional growthshould think twice before adopting the curriculum. If they do want to adopt it, they should consider providing intensive professional development and support to assist teachers in achieving high implementation.

Our study also leaves a number of questions unanswered, which we suggest should be addressed by further research. For example, can quality of implementation be improved through professional development and support? If so, what kind of professional development and support is needed? These questions are critically important to address as increasing numbers of schools and districts are attempting to adopt learner-centered teaching approaches in response to more rigorous Common Core Standards. Findings from this study suggest that curriculum adopters should be careful to ensure strong implementation of those curricula.

Acknowledgments

This research was conducted by the RAND Corporation with funding from the Institute of Education Sciences (R305F050122).  The opinions expressed in this article are those of the authors and do not necessarily represent the views of the sponsor.  We would also like to acknowledge the contributions of project team members Dahlia Lichter, Daniel McCaffrey, Mary Slaughter, Dan Gershwin, and Jennifer Novak.

Notes

¹ The district as a whole enrolled 107,043 students in the 20052006 academic year (National Center for Education Statistics, 2008).

² Extensive detail about the randomization design, sample balance, and attrition is provided in Pane et al., 2010. As a randomized trial with low overall and differential attrition, the study has been determined to meet What Works Clearinghouse evidence standards without reservations (What Works Clearinghouse, 2013).

³ An important advantage of randomizing within teachers is that we eliminate between-teacher differences in average effectiveness as a source of noise in our treatment-effect estimates, thereby improving our statistical power for identifying a treatment effect. A tradeoff of this design is the potential for contamination, in which teachers may adopt aspects of the treatment in their control classrooms, thereby biasing the treatment effect estimate toward zero. Our qualitative data provide an indication of the teaching practices used in treatment and control classrooms, and those practices are the primary focus of this chapter.

⁴ Statistical models, estimated coefficients, and standard errors are reported in Pane et al. (2010).

References

Anderson, J. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.

Ball, D. L., & Cohen, D. K. (1996). Reform by the book: What isor might bethe role of curriculum materials in teacher learning and instructional reform. Educational Researcher, 25(9), 68, 14.

Brown, S., Pitvorec, K., Ditto, C., & Kelso, C. (2009). Reconceiving fidelity of implementation: An investigation of elementary whole-number lessons. Journal for Research in Mathematics Education, 40(4), 363395.

Chávez, O. (2003). From the textbook to the enacted curriculum: Textbook use in the middle school mathematics classroom (Unpublished doctoral dissertation). University of Missouri-Columbia, Columbia, MO.

Chval, K. B., Chávez, Ó., Reys, B. J., & Tarr, J. (2008). Considerations and limitations related to conceptualizing and measuring textbook integrity. In J. T. Remillard, B. A. Herbel-Eisenmann, & G. M. Lloyd (Eds.), Mathematics teachers at work: Connecting curriculum materials and classroom instruction (pp. 7084). New York, NY: Routledge.

Creswell, J. (1998). Qualitative inquiry and research design. Thousand Oaks, CA: Sage.

Desimone, L., Smith, T., Baker, D., & Ueno, K. (2005). The distribution of teaching quality in mathematics: Assessing barriers to the reform of United States mathematics instruction from an international perspective. American Educational Research Journal, 42(3), 501535.

Educational Testing Service. (2004). Administrators manual for the assessment of algebraic understanding (No. 724869). Princeton, NJ: Educational Testing Service.

Eisenhart, M., & Howe, K. (1992). Validity in educational research. In M. LeCompte, W. Millroy, & J. Preissle (Eds.), The handbook of qualitative research in education (pp. 643680). San Diego, CA: Academic Press.

Glaser, B. (1978). Theoretical sensitivity: Advances in the methodology of grounded theory. Mill Valley, CA: Sociology Press.

Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.

Hord, S., Stiegelbauer, S., Hall, G., & George, A. (2006). Measuring implementation in schools: Innovation configurations. Austin, TX: Southwest Educational Development Laboratory.

Huntley, M. (2009). A brief report: Measuring curriculum implementation. Journal for Research in Mathematics Education, 40(4), 355362.

McLaughlin, M. W., & Mitra, D. (2001). Theory-based change and change-based theory: Going deeper, going broader. Journal of Educational Change, 2, 301323.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.

National Center for Education Statistics. (2008). Local education agency (school district) universe survey, 2005-06, v. 1a. Washington, DC: U.S. Department of Education. Retrieved March 10, 2015, from http://nces.ed.gov/ccd/elsi/quickFacts.aspx.  

National Research Council. (2004). On evaluating curricular effectiveness: Judging the quality of K12 mathematics evaluations (J. Confrey & V. Stohl, Eds.). Washington, DC: National Academies Press.

Pane, J., McCaffrey, D., Slaughter, M., Steele, J. L., & Ikemoto, G. (2010). An experiment to evaluate the efficacy of Cognitive Tutor geometry. Journal of Research on Educational Effectiveness, 3(3), 254281.

Remillard, J. (2005). Examining key concepts in research on teachers use of mathematics curricula. Review of Educational Research, 75(2), 211246.

Ritter, S., Kulikowich, J., Lei, P., McGuire, C., & Morgan, P. (2007). What evidence matters? A randomized field trial of Cognitive Tutor Algebra I. In T. Hirashima, H. Hoppe, & S. S.-C. Young (Eds.), Supporting learning flow through integrative technologies (pp. 1320). Amsterdam, The Netherlands: IOS Press.

Schoenfeld, A. (2006). What doesnt work: The challenge and failure of the What Works Clearinghouse to conduct meaningful reviews of studies of mathematics curricula. Educational Researcher, 35(2), 1321.

Tarr, J., Reys, R., Reys, B., Chávez, Ó., Shih, J., & Osterlind, S. (2008). The impact of middle-grades mathematics curricula and the classroom learning environment on student achievement. Journal for Research in Mathematics Education, 39(3), 247280.

What Works Clearinghouse. (2013). WWC intervention report: Carnegie Learning curricula and Cognitive Tutor®. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

APPENDIX A: SUMMARY EXAMPLE OF A HIGH IMPLEMENTING COGNITIVE TUTOR GEOMETRY CLASSROOM

Lesson Introduction. After spending seven minutes reviewing the homework, the teacher introduced the Cognitive Tutor lesson on area of a triangle by having students turn to page 21 of their workbooks and reading the scenario at the top of the page: In sailboat races, one of the typical shapes of a racing course is triangular. The course path is identified by buoys called marks. When the course is a triangle, the marks are located at the vertices of the triangle. The teacher helped students make connections to their life experiences by asking, Where have you seen sailboats? Students called out responses such as, At the marina and We saw them in Old Town. She reviewed vocabulary like racing course and buoys and made connections to students prior knowledge about vertices.

The teacher also made connections to future material by saying, The wind has a lot to do with where the boat goes. We will learn more about wind when we study vectors. She made more connections to prior knowledge by telling students to recall what they learned about parallelograms and to apply that knowledge to solving for the measures of a triangle. To help students solve for the area, she reminded them of an activity they had undertaken in the first week of school. She prepared students to address potential problems by telling them that they would need to know the formula for the area of a triangle, reminding them of the formula and how to identify the height. She further reminded them that height does not always go up and down, and showed them an example of a triangle for which the height was horizontal. Finally, she asked students to turn to a question on page 26 and to realize that when the text asked them for the length of the course, it was really asking them to solve for the perimeter.

Activity and Materials. The students were already seated in clusters of four desks with their regularly assigned group members. The teacher told students to work with their group members to answer questions on pages 21 and 22 of their workbooks, which consisted of nine open-ended questions, such as: In sailboat races, one of the typical shapes of a racing course is triangular.  Estimate the area enclosed by the course. Use a complete sentence in your answer.

Teacher Practice. As the students began working in their groups, the teacher encouraged them to be responsible for their own learning by carefully re-reading the problem, using their glossary, and consulting with team members before asking the teacher for help. As she circled the room, she almost always responded to students questions by prompting them to ask another group member. In one instance, she re-directed the question herself. As a result, she typically only spent a few seconds responding to any one student. When she thought a question warranted further explanation, she stopped the entire class to provide an explanation at the board.

The teacher frequently encouraged students to be responsible to their groups. For example, she told one student that he was working too far ahead and that he should wait for his team members. She also reprimanded a student who suggested to his group that they divide up the problems, telling him, You know we dont do that! She also announced to the class that it was more important to have the same answer than to have the right answer.

Student Engagement. The students spent 30 minutes working in groups. Although there were many examples of students asking the teacher for help, the observers witnessed multiple examples of students explaining how to solve a problem to their group members. For example, one student showed another how to find the length of a side by counting along the grid line. In another group, a student asked a group member, What did you get for question 5? After her group member responded, she asked, Wait, how did you get that? and her group member explained her process for arriving at the solution.

Although there were several instances of students socializing during group work time, they often continued to work while socializing or only remained off task for a short period of time. All students were engaged for the vast majority of the period.

Closure. The teacher brought closure to the lesson by having each group present its answers to two of the workbook problems. She gave them 10 minutes to write their answers on the board before the presentations began and encouraged students to be responsible for one anothers learning by telling the rest of the group members to Pay attention to what your group member is writing and let them know if you dont like something theyve written so that they have time to correct it. She reiterated this by walking over to a group that was not paying attention, telling them to look at what their group member was writing on the board, and instructing them to make sure it was right.

She prefaced the actual presentations by reminding students that they were responsible for paying attention and for understanding how to solve all of the problems. She said they should have all the right answers on their paper by the end of the period, but that they should refrain from copying down answers until the problem had been discussed and they were certain that the answer on the board was correct.

She required groups to assign a presenter who had not presented before, but reiterated that the group was responsible for making sure their presenter understood the problem and presented it correctly. When presenters made mistakes or could not answer the teachers probing questions, she re-directed the questions to the group and then to the entire class. She also allowed time for the rest of the class to ask the presenters questions. By the end of class, the students had reviewed all of the problems as a classallowing the teacher to assess and reinforce their understanding of the material.

APPENDIX B: SUMMARY EXAMPLE OF A TYPICAL TREATMENT COGNITIVE TUTOR GEOMETRY CLASSROOM IN BCPS

Lesson Introduction. After spending 12 minutes on a drill and reviewing it as a class, the teacher introduced the lesson by having the students turn to page 157 of their workbooks and asking a student to read the scenario out loud. As the student read, the teacher addressed three students who had not come to class prepared with their workbooks. The class waited and began to socialize as the teacher took four minutes to access the Carnegie Learning website to print extra copies of the workbook pages for the students without textbooks. No other introduction was provided before the teacher assigned the activity.

Activity and Materials. After a few moments spent regaining the students attention, the teacher explained that they would be working on pages 157 and 158, and that the assignment should be completed by the end of the class period. She assigned new groups, explaining to the students that they had not worked well together in the previous groups and that the new groups were designed to provide better matches. The assigned pages consisted of a scenario and six open-ended questions, similar to the assignment in Appendix A.

Teacher Practice. The teacher circulated among the groups, fielding students questions. She repeatedly encouraged students to pose questions to one another and explain answers to group members when they figured them out. For example, she responded to the first three questions she received by saying, Ask a group member. She further discouraged students from relying on her for answers by providing each group with two question cards. Each group had to turn in a question card to ask the teacher a question, and therefore was limited to two questions per class. The teacher began accepting question cards and sat with each group for several minutes, helping them to work through the problems. She also approached groups when she noticed they were off task to encourage them to refocus on their work.

Student Engagement. At the beginning of the group work activity, students took approximately 10 minutes to physically relocate themselves into their groups, and once there, many socialized or simply stared into space until they were prompted to engage by the teacher. After another four minutes, most of the students were actively doing the work, but were working independently rather than with peers.

For each group, the observer noted only one or two exchanges in which students were discussing mathematics. Though the teacher had prompted students to pose questions to one another, they were not responding accordingly. For example, when a student called the teacher over with a question, the teacher asked the other members of the group whether anyone had determined the answer. One student replied with the correct answer. The teacher acknowledged it was the right answer, but said, Explain how you got that to your group; dont just give the answer. However, when the teacher walked away, the group began to socialize instead.

The observer noted additional instances of weak student engagement. For example, after 10 minutes of group work, two members of one group had not written anything on their papers. The other two members of the group had written answers to three problems, but were staring into space as the observer walked by them. The observer noted that the latter two students re-engaged four minutes later when they began discussing their answers to the second problem. In another group, two boys were sitting close together; one was quickly answering many of the problems while the other directly copied the answers onto his own paper.

Closure. The class ended when the bell rang while students were still arranged in groups. It appeared that some students had finished the assignment but most had not. As students exited, the teacher announced that any unfinished work should be completed for homework.

APPENDIX C: SUMMARY EXAMPLE OF A TYPICAL CONTROL GEOMETRY CLASSROOM IN BCPS

Lesson Introduction. After spending 25 minutes at the front of the room leading students through a check of homework questions and through the steps to solve a warm-up drill, the teacher began the days lesson by writing on a transparency projected onto the board and telling students to take notes. She wrote the objective, Students will be able to name similar parts of congruent figures and find the similarity ratio of similar figures and then drew two congruent triangles and explained how to label them. After each step, she waited for students to copy the material. She also explained how to determine which sides were congruent and then quizzed the class regarding particular sets of sides. For example, she asked, Can you say that CA is congruent to FD? Students responded yes or no. Then she drew two sets of trapezoids, asked students to define trapezoids, and used a question-and-answer technique to prompt students to name corresponding angles and sides.

Next, the teacher introduced the concept of similar figures. She asked if they had learned about similar figures before, but no one answered this question. She wrote the definition of similar figures on the boardThe angles are congruent and sides are proportionaland asked where they had seen the term proportional before. When no one answered, she told them they had seen it in pre-algebra and tried to jog their memories by listing the names of pre-algebra teachers at the school.

The teacher then defined the similarity ratio and demonstrated how to solve for it. She used a question-and-answer technique to walk the class through several examples, which included the following exchange:

T: What side corresponds with y?

S: 6

T: Right, so I set my similarity ratio equal to y over 6. Make sure that if your similarity ratio is for big triangle over small triangle that you are setting your new fraction so that the big triangle side is over the small triangle side. So, 2 over 1 equals what?

S: y over 6

T: Right and then you cross multiply. So, you get y = 12.

The teacher attempted to get the students to lead the next problem by asking for someone to set up the similarity ratio, but no one answered. She called on a student who provided the wrong answer; the teacher then led the class in solving that problem and the following problem.

Activity and Materials. After the 30-minute lecture, the teacher distributed a ditto that she had created with 21 practice problems related to similar parts of congruent figures and similarity ratios. For example, one question in reference to a pentagon diagram was, If AB = 6.3 inches, find FG. She told students that they could opt to work with a neighbor.

Teacher Practice. During the first part of the practice problems, the teacher responded to raised hands. She typically sat with each student for several minutes, walking the student through the problem step-by-step. The following exchange was typical:

T: You need to name this triangle;, what do you want to name it?

S: [Responds by labeling the triangle vertices A, B, and C]

T: Good. Now you need to name the second triangle, but order matters this time. If you started with this [pointing to angle of first triangle], which angle do you need to start with on the second triangle?

S: B

T: Right, and which angle should you name next?

After the teacher spent several minutes explaining the first problem to three different students, she pulled the class back together as a group, and used a transparency to walk the entire class step-by-step through the next three problems. Then she instructed students to finish the rest of the practice problems on their own, and she continued circulating through the room.

Student Engagement. During the lesson introduction, the majority of students were following along and actively taking notes. However, two students did not have a notebook out and a third student was sleeping with his head on the desk. There were only two or three students who voluntarily responded to questions when the teacher posed them to the class.

During seatwork, the majority of the students remained seated individually. Two girls moved their desks closer together and socialized quietly. Two boys moved their desks together but appeared to continue working individually. At first, the room was quiet. A couple of minutes after starting the assignment, the observer counted six hands in the air, indicating these students were waiting for the teacher to provide assistance. After a few minutes, these and several other students began socializing as they waited for the teacher. When the teacher called them back together as a class, the majority of students copied the answers as the teacher walked through the problems. When they returned to individual work, the majority of students continued to work on the problems, although some appeared to be working more diligently than others. Most student interactions were social in nature, including a lot of bantering and teasing. In a few instances, the observer noted one student asking another for the answer to a particular problem, as in, What did you get for number 7? But in these instances, students did not discuss how they had arrived at their answers.

Closure. The class ended when the bell rang. Approximately 10 minutes before the end of class, several students had finished the assignment, and the teacher suggested they start their homework, but they socialized instead.

Cite This Article as: Teachers College Record Volume 118 Number 13, 2016, p. 1-34 https://www.tcrecord.org ID Number: 20563, Date Accessed: 10/26/2021 12:49:49 AM Purchase Reprint Rights for this article or review