A Study of Comprehensive School Reform Programs in Arizona

Conclusions/Recommendations: School reform initiatives can profit from more research on participant perceptions, actual classroom practices, and student mediation of those practices. These understandings can better link program design and student achievement to enhance the effectiveness of comprehensive school reform initiatives.

Increasingly, more American youth live in poverty and attend schools with few resources relative to schools that serve affluent students. The federal government has addressed this inequality through Title I initiatives that offer new resources to education. Title I programs have made tremendous investments to support the learning of students living in poverty who attend underresourced schools. Research findings on the impact of Title I on student achievement are mixed but appear modest at best (Slavin, 1999). Disappointment with these results has fueled a change in Title I strategy leading to the creation of the Comprehensive School Reform Demonstration (CSRD) Program, subsequently referred to as CSR. CSR is whole-school or schoolwide reform designed to enhance student achievement and sustain it over time. Later in this article, we elaborate this definition of CSR and describe research extant on CSR at the start of our research project.

In the fall of 2000, we were awarded a grant to study the effects of CSR on student achievement in Grades 3–5 in the state of Arizona. This article describes our research plan and discusses how our research goals changed over the course of the grant because of unexpected constraints and opportunities. This information frames the study and its emerging context; however, it also provides useful “lessons” for those who wish to study school reform. In the end, the fundamental lessons we learned were to recognize the central role of poverty, the discontinuity that comes with mobility, and their combined negative impact on learning in classrooms.

THE PROBLEM

When we started the project, it was evident that the number of youth living in poverty was growing rapidly (see, e.g., Federal Interagency Forum on Child and Family Statistics, 1999; National School Boards Association, 1999). Poverty interferes with educational performance (Ladd & Hansen, 1999). Typically, children of poverty attend schools with the fewest financial resources. Compelling data illustrate that states and school districts have unequal resources and that generally, schools serving low-income students receive fewer funds than do schools that educate students from more affluent homes. In some instances, even in the same district, schools that serve less affluent constituents receive lower funding than do other district schools (Ladd & Hansen; Stiefel, Rubenstein, & Berne, 1998). Hence, many American youth face both educational and home poverty.

In the past, some social scientists argued that school expenditures were largely unrelated to student performance (Averch, Carroll, Donaldson, Kiesling, & Pincus, 1974; Coleman et al., 1966; Jencks et al., 1972). Some educators continue to make this argument (Hanushek, 1997) but with a different twist. The accepted explanation for the alleged failure of school expenditures to make a difference in the achievement of students living in poverty has changed from the genetics of home arguments of the 1960s (e.g., Jensen, 1973) to the economics of home arguments presently. Alleged causation apparently has shifted from “nature” to “nurture.” In each perspective, however, the cause and effects are treated as unidirectional, suggesting the futility of increased resources for schools that serve students of poverty (see also McCaslin, Burross, & Good, 2005).

Both the logic and empirical data associated with the Coleman report (Coleman et al., 1966) were assailed by researchers writing in the 1970s (Good, Biddle, & Brophy, 1975; Klitgaard & Hall, 1974; Wise, 1976, as cited in Ladd & Hansen, 1999). More recent and sophisticated studies have reached more positive conclusions about the relationship between resources and results (see the Ladd and Hansen's 1999 National Research Council publication Making Money Matter) and suggest the possibility that when poorly funded schools receive more funding, they may improve their instructional resources for students. If they use those resources wisely, student achievement can be enhanced (see, e.g., Biddle, 1997; Ladd & Hansen).

Results need not be dramatic to be important. For example, Rowan, Bossert, and Dwyer (1983) agreed with the input-output researchers of the 1970s that schooling accounts for a relatively small percent of variation in student achievement; however, they also contended that such effects are important, especially over time. There is reason for guarded optimism. However, poorly funded schools are impoverished on more than monetary measures. Many teachers are unwilling to teach, let alone live, in certain neighborhoods, and some teachers possess an inadequate knowledge base to frame instruction meaningfully for students of poverty.

In short, for multiple reasons, schools that serve students primarily from low-income homes, on average, are less prepared to meet the educational needs of their students as compared with schools that educate more affluent youth. This is especially detrimental in the early grades. It has been asserted since at least 1973 that student performance in third grade (especially reading performance) largely predicts student performance in high school and beyond (e.g., Klaus, 1973). Nonintervention in underresourced schools serving children of poverty is not an option—unless as a society we choose to leave these children unprepared for productive lives.

REVIEW OF THE LITERATURE¹

EFFECTIVE SCHOOLS

Much of the early “school effects” research relied on group average data, masking what might be important variations among schools (Klitgaard & Hall, 1974). Profiles of individual schools (e.g., outlier schools), rather than averaged school profiles, yielded consistent differentiations between more and less effective schools even when controlling for student characteristics (Brookover, Beadie, Flood, Schweitzer, & Wisenbaker, 1979; Rosenholtz, 1989; Rutter, Maughan, Mortimore, Ouston, & Smith, 1979; Weber, 1971). Naturally occurring effective schools could be found, and in the 1980s, there was growing consensus on what they looked like (Good & Brophy, 1986). Effective schools were well managed, and their content and performance goals were coordinated to direct resources to student achievement. The effective school research in the 1980s was based primarily on correlational studies; however, there were some experimental research attempts to improve school-level performance, with mixed results (e.g., McCormack-Larkin & Kritek, 1983).

Effective schools research continued into the 1990s. Teddlie and Stringfield (1993) conducted a notable longitudinal study of effective schools in Louisiana over a decade. They identified some aspects of school expectations for teachers and students that “allow” teachers to be more effective. On the basis of several paired comparisons, Teddlie and Stringfield also noted four areas in which school effects appeared important: selection and replacement of teachers, classroom monitoring and feedback, support for improvement of individual teachers, and allocating and protecting academic time.

EFFECTIVE TEACHERS

We mention the teacher effectiveness literature here even though we studied whole-school reform because many aspects of the effective teacher literature—for example, proactive management, and active and explicit teaching—have been consolidated into many (but not all) CSR programs. It is reasonable to assert that what teachers do while teaching is a major determinate of how well students learn. What Good et al. (1975) had argued early on, primarily on logical grounds, now has become commonly accepted knowledge in the field.

TITLE I RESEARCH

The federal government has been interested in improving the school performance of students who come from low-income homes since the time of Lyndon Johnson’s “Great Society” initiatives in the 1960s. Title I has a complex history, and the many attempts to evaluate its effectiveness suggest the importance of an integrated curriculum, quality of program, and fidelity of implementation in improving student achievement (Borman, Stringfield, & Slavin, 2001; Cohen, 1983; D’Agostino, Borman, Hedges, & Wong, 1998; Ross, Alberg, & Nunnery, 1998; Stringfield et al., 1997; Wong & Meyer, 1998). Borman and D’Agostino (1996) conducted a comprehensive meta-analysis of extant studies and concluded that Title I is effective to the extent that schools strategically use resources over time. These authors also noted that Title I effects appear to have plateaued; however, Title I schools outperformed control schools in enhancing student achievement with a d = .11 effect size.

COMPREHENSIVE SCHOOL REFORM DEMONSTRATION PROGRAM INITIATIVE

The U.S. federal government strongly supports the belief that good schools can be created and has responded to mixed and modest gains in the achievement of students living in poverty with increased funding and changed strategies. In 1994, Title I was reauthorized to allow schools to implement a whole-school reform model for all their students, not only for those of low-income families who typically received supplemental instruction. This change recognized both the reality of underresourced schools that serve large numbers of students of poverty, and the potential in targeting the school rather than the student. During 1998 and 1999, Congress and the Clinton administration agreed on a new definition of federal involvement in education in the form of the Comprehensive School Reform Demonstration Program (CSRD) and the Reading Excellence Act (REA). The CSR program brought considerable new funds to schools that promised to use “good” research-based practices to design and coordinate—or purchase—program reform components. Hence, CSR models were influenced more by the school effectiveness research than by Title I research.

The 11 CSR components that have been used as critical guidelines for implementing whole-school reform are (1) effective, research-based methods and strategies; (2) comprehensive design with aligned components; (3) ongoing professional development; (4) measurable goals and benchmarks; (5) support within the school; (6) support for teachers, principals, administration, and other staff; (7) parent and community support and involvement; (8) external support and assistance; (9) evaluation strategies; (10) coordination of resources; and (11) scientifically based research (the latter two were added subsequently). For expanded explication of these criteria, see Borman, Hewes, Overman, and Brown (2003).

COMPREHENSIVE SCHOOL REFORM PROGRAMS

CSR reform is whole-school or schoolwide reform designed to enhance student achievement and sustain it over time. CSR is largely informed by school effectiveness literature reviews (see Good & Brophy, 1986; Teddlie & Stringfield, 1993). Exciting design elements are embedded in CSR programs; however, their organization in practice is less than coherently integrated. For example, a typical CSR program “assembles” an array of current school reform ideas (e.g., parent involvement); earlier research on teachers (e.g., teacher expectations) and recent research on various specific classroom practices (e.g., cooperative learning) or new conceptions of subject-matter learning (e.g., project-based learning); and advances in knowledge about how to teach subjects such as math and reading. Some new or popular classroom instructional programs have mixed empirical support, however, and may not be associated with gains in student achievement in schools that serve students living in poverty. For example, there are ample data suggesting that extensive use of cooperative group work (especially without careful consideration of the quality of group process) is not a panacea for increasing student motivation and achievement (McCaslin & Good, 1996a). Given the level of personal readiness and resources that this format presumes, it seems especially important to consider issues of quality in group process when using cooperative group work with students who live in poverty. Yet many reform programs—CSR included—recommend such practices with little or no qualification.

CSR EFFECTIVENESS

In some respects, the results of research on the effectiveness of CSR parallel earlier results on the impact of Title I in that many initial CSR studies have shown positive effects, whereas many others have not. However, sometimes it takes years before effects occur. For example, Bloom (2002) studied the effects of the Accelerated School Program on average third-grade reading and math scores. The schools that Bloom studied began using the Accelerated School Program in the 1990s. Program effects were estimated by comparing the overall mean scores for baseline—3 years that preceded treatment implementation—relative to mean scores for 5 years of posttreatment school performance. There were no program effects for the first 4 years of implementation, and only minor effects were observed in the 5th year. Effects were stronger in mathematics than in reading. One important aspect of this study is that the authors could estimate where the achievement gains had occurred—which students improved—over the intervention period. Bloom found no achievement changes over time in the students who scored in the lower baseline category, typically the target student in school reform. The positive effects in Year 5 were the result of some average-performing students (25–75 quartiles) moving up into a higher quartile. Thus, the 5th-year effects of the intervention primarily were due to the improved performance of a subgroup of average-achieving students. There were no general effects across the entire distribution of students, nor were there improvements among the low-achieving students, the primary target of school reform.

Clearly, understanding CSR effectiveness is complex. Achievement gains may occur in some subjects more than others, and program effectiveness can vary with student achievement level. In a comprehensive review of extant data, Borman et al. (2003)² compared the effects of 29 commonly used CSR programs on student achievement. They reported that although some program effects were large, on average, the effects of CSR programs on student achievement were similar to those found in earlier Title I programs. In all CSR studies that used a control group, the effect size of the CSR programs was d = .12. Further, in 55% of the comparisons with the control group, CSR schools outperformed control schools.

OUR PROGRAM OF RESEARCH

We have described CSR programs and their effects on student achievement. We now turn to a description of our project and our attempts to strengthen research on schools that serve students living in poverty and that are engaged in school reform.

A NEED FOR THEORY

Research on CSR reform programs typically involves a comparison of a school implementing a CSR model with a control school. However, even if programs consistently had positive impacts on student achievement, without a theoretical framework, it would be problematic to argue that the program per se impacted achievement. We believe it is this lack of theory that has made the transition of basic research findings into models of practice problematic. Results are packaged for the use of key elements, but without a way to think about context and rationale, they remain just that: behaviors that may not be suitable in a new context or that are not always aligned or pointed in the same direction. We wanted to bring theoretical concerns to CSR, and we asked, How can theories of student motivation and learning inform the effectiveness of CSR programs?

ANCHORING OUR RESEARCH: FOURTH GRADE WINDOW

There has long been considerable debate about the achievement deficiencies and needs of students who come from low-income homes. Although when we began our research, there were data to suggest that CSR programs could enhance student achievement, program effects seemed likely to be small, and positive effects were not always found. CSR programs (and earlier Title I programs) are interventions designed to address the achievement deficiencies of students living in poverty. Some have argued that these programs have missed key variables: namely, the needs of students living in poverty.

Pogrow (1999), for example, argued that a misunderstanding of the learning needs of Title I students beyond third grade leads to an inappropriate curriculum. Pogrow wrote,

The biggest gains and the biggest reduction in gaps in Title I occur at the very early elementary grade levels. These gains decline, and gaps increase, thereafter and by the time students enter junior high, the gains have dissipated. . . . Simply put, grades 4–8 are the black hole of American education which seems to suck up whatever progress has been made earlier and where things seem to fall apart. Hence, one argument for the widespread failure of students is because the instruction they receive is insufficient to prepare students for exams that are more cognitively difficult.

Although Pogrow’s “cognitive wall” hypothesis (i.e., the thinking demands of an increasingly complex curriculum overwhelm students’ ability to learn at a particular point in time) is plausible, it is certainly not the only hypothesis that might explain significant and rapid drops in student performance. Alternatively, we posited the “fourth-grade window” as a central construct in student-motivated learning in our research. We believe that factors other than, and in addition to, learning the curriculum mediate student performance deficits in the fourth-grade window. In particular, we focus on students’ conceptions of school, learning, and self.

Students in Grades 3–5 experience more than a changed curriculum and the cognitive demands it requires. In these grades, peers take on new significance, and fitting in socially can present new and sometimes enduring challenges. Typically, this is also the time when parent influence on children’s learning in school begins to diminish (e.g., after the initial years of schooling, parents’ presence in schools drops off markedly). In this interpersonal context, students’ identity, their sense of self that is rooted in social relations, becomes increasingly important as it solidifies over time and across situations (McCaslin & Good, 1996a, 1996b).

PROJECT EXPECTATIONS

The most basic goal of our study was to inform working theories of learning, motivation, and social/emotional development in school contexts in Grades 3–5. We expected to inform theoretical arguments that underlie achievement expectations in CSR models.

Implementation—classroom practice—is everything if we are to understand what happens in the lifeline of comprehensive school reforms. We hoped that an emphasis on theory, contextual enactment, and participant mediation of classroom practices would yield a fuller picture of instructional practices and the motivational dynamics of students in CSR schools, and how these factors compare with normative (typical) practices and beliefs. We also had hoped to include teachers’ beliefs and preferences, but we could not, for reasons we explain later.

RESEARCH CONTEXT

We conducted our research in the state of Arizona. Nationally, Arizona continually ranks exceedingly low on most indicators of support for children, especially children of poverty. Arizona has much to learn about the amelioration of poverty through education. It is imperative that schools adopting CSR models be studied carefully to learn how to increase these students’ achievement. It likely is only through student achievement gains that policy makers in states such as Arizona will change their beliefs about the power of education and be more willing to increase educational funding, which can enrich the quality of life of disadvantaged students.

We believe that schools can make a difference in enhancing student achievement while taking into account the constraints of home and community resources. As we have argued (McCaslin et al., 2005), it appears unrealistic to suggest that schools can eliminate the effects of socioeconomic status (SES) and transform students from low-income homes into students from high-income homes, especially given the way schools are configured and funded. This is not a call for a return to the arguments of the 1960s, asserting that schools do not impact student achievement; rather, it is a call to recognize the power that schools may well have to make a difference in the lives of students living in poverty if the constraints on that difference are recognized. To do otherwise—to suggest that schools can transcend the home and community of the students they serve—is to ensure that schools serving students of poverty will fail. Indeed, to do otherwise is to equate poverty students with low-ability students, low-performing schools with “failing” schools—which some policy makers already do. And failure sells. The media readily disseminate depictions of the problem of the failing public school.

CHANGES IN SCOPE OF WORK

ARIZONA STATE DEPARTMENT OF EDUCATION

Our project was funded in September 2000. The first funding of CSR schools in Arizona was in 1998. Because these schools had been implementing their programs for 2 years, we reasoned in our proposal that it would be more productive to begin data collection with a new wave of schools. It was expected that new CSR schools would be funded during the 2000–2001 school year. This did not occur. No new CSR schools were funded until 2002, first in January and then again in August. The delay in funding had a negative impact on the planned research because sample recruitment and data collection were delayed for over 18 months. This time would have been used to establish rapport with teachers and students because more personal forms of data collection (interviews, diary formats) had been planned.

Another issue that impacted project research plans was the fact that when awards finally were made in January 2002, only one K–8 school located in or near Tucson received CSR funding. In August 2002, no CSR-funded K–8 school was in the Tucson area. Thus, most funded schools were at least 100 miles from the home base of our research project at the University of Arizona, and some were considerably further than that from Tucson. The location of the CSR schools added to the cost of data collection, and the long drives made it difficult for staff to conduct informal visits (before or after school, during lunch, and so forth). Informal visits would have enhanced formal observations and would have allowed more opportunities to capture teacher perspectives of CSR reform. As we learned subsequently, however, teachers also had transportation concerns, and their time available outside school hours was constrained in many sample schools.

TEACHER PARTICIPATION

The need for teacher “buy-in” has been long recognized in the school reform literature, and we learned that it also applies to researchers who want to study reform. Building rapport with teachers and finding time for formal and informal interviews was much more difficult than we anticipated. Teachers had little discretionary time during the day, and typically they did not live in the school district or community where they taught. Many teachers left directly at the end of the school day to avoid traffic hassles and delays associated with commuting—for some as much as 100 miles. Our distance from schools significantly added to the problem and left little time for our coders to do anything other than observe classes. Math and reading are typically taught in the morning. We did not have discretionary time to stay after school to see what students did in after-school programs or to visit their surrounding community.

CONTACT WITH PRINCIPALS

One positive impact of the delay in funding, however, was our increased contact with principals. Principals generally were willing to share their knowledge of comprehensive school reform and grant us access to their schools. Ironically, this was due in part to the delay in funding: All of us were waiting on the Arizona Department of Education (ADE). Project staff attended various meetings in the first 18 months of the project, along with principals and members of their faculty. ADE required attendance at several meetings if schools wanted to compete for CSR funding. ADE supported our attending them as well. Meeting topics included how to make school reform work, how to write a CSR proposal, and new CSR criteria. There also were several optional meetings, such as vendor fairs where representatives of CSR programs explained the important features of their programs and answered questions.

During these meetings, we had the opportunity to observe the same activities as principals or their representatives. We had informal conversations during breaks and lunches. One new dimension in the 2002 funding (as compared with the 1998 funding) was the requirement that each competing school hire a facilitator to help them write their CSR proposal. As a part of the application process, schools had to write a preproposal, and on this basis, if successful, schools were awarded a planning grant of $35,000. These funds could be used in part to hire the facilitator/consultant who then helped the school write its CSR application. We also attended training meetings for individuals interested in becoming a school or program facilitator, and we met several people who eventually become facilitators. Successful CSR applicant schools ultimately received $50,000 from ADE for each of 3 years, totaling $150,000.

In sum, by the time the new funded CSR schools were announced, we had met many of the principals and some facilitators. In cases in which we did not know them, we knew the reform language and activities they had experienced. We were “trained” in school reform at meetings at which we also were referred to as “you people,” we heard the same tired jokes, and we struggled with the same pretest knowledge of school reform as did school personnel. This knowledge helped us communicate our interest in studying the reform process and convey the message that we knew some of the constraints and obstacles they faced. When we were recruiting schools to participate in our research—and most directly, principals—the pressures of high-stakes testing and No Child Left Behind were clearly manifest. The language of failing schools was in the media, and high demands were generated—and repeated frequently—for large and immediate gains in student performance on mandated achievement tests. Therefore, the idea of formal research being conducted in their school—including classroom observation—in the midst of their emerging CSR plans and implementation likely was not immediately appealing to many CSR principals.

RECRUITING SAMPLE SCHOOLS

We first sent information to each funded school explaining who we were and why we were contacting them. Shortly after that, principals were called by one of the co–principal investigators (Tom Good), who expressed interest in following their reform work and requested the opportunity to visit with them personally. Meetings took place in the principals’ offices. Some principals agreed to participation without staff input; others indicated their school would participate only if teachers consented to the project. Still other principals agreed to participate, but only if their school districts supported it. In those cases, another meeting with the school superintendent was required. In one case, a formal proposal had to be presented to the school board.

Principals likely agreed to the research for several reasons. First, there was some expectation that funded schools should participate in research on CSR, and for some, this was sufficient. Principals were reminded of this expectation; however, we did not stress it because we had no interest in mandating participation. Second, some principals were new to their jobs and wanted a chance to learn others’ opinions about and experiences with school reform.

Principals’ support, especially their interviews, pulled us away from an earlier focus on teacher mediation of CSR programs to a fuller systems-level policy examination. Through the principals’ eyes, we saw reform as including more than classrooms. We came to see ways in which the community, media reports, parents, school district support (or lack thereof), and much more mediated reform progress.

CHANGES IN CSR PROGRAMS

As part of our project, we also attended required conferences sponsored by the Office of Educational Research and Improvement (OERI) that brought together CSR researchers from across the nation to discuss common problems and opportunities, and concepts and findings. One important discovery for our project and our project officer was that CSR was becoming a moving target. Initially CSR funds were awarded to programs that enacted a single comprehensive reform model. But the conception of CSR began to change in important ways. Schools began submitting, and states began funding, programs that selected features from different models. These programs became known as “hybrid” or “home-grown” programs. In addition, the number of models available for schools to draw from increased because more flexible criteria were being used to define research evidence.

PROJECT AND OERI NEGOTIATED CHANGES

In consultation with our program officer and review panel, a decision was made to change from a comparison group design to a study of only CSR schools. We decided to observe more intensively in some schools than others and to conduct case studies in a handful of schools. We planned to collect enough classroom data across the entire sample to see how CSR instruction compares with normative instruction nationally (see Wiley, Good, & McCaslin, 2008). We also decided to conduct a participant observation study in one school to inform instrumentation to assess students’ motivation dynamics, their understanding of school, learning, and sense of self (see McCaslin, 2008). These instruments were then administered in case study schools that represented three school performance labels—underperforming, maintaining, and improving—in the state.

RATIONALE FOR THE CHANGES

Some changes in our research were due to the impact of the delayed timeline in ADE identifying new CSR schools. Other changes were made because of the increasing knowledge of, and database on, CSR schools. As noted, research studies and even a meta-analysis of CSR effects on achievement in comparison with control schools had provided a good base for estimating the likely impact of CSR programs on students’ achievement (Borman et al., 2003). OERI and an external review panel felt, and we agreed, that our project might have more impact in its qualitative aspects, particularly to develop a better understanding of student-motivated learning.

NEW CONTRIBUTIONS

The delay in funding of CSR schools allowed at least two things to occur that would not have been possible otherwise. While waiting for the award process to be completed, we deployed personnel in other ways. For example, we collected and analyzed the effectiveness of the 1998 CSR-funded schools in Arizona on student achievement over time (Good, Burross, & McCaslin, 2005). This provided a valuable comparison of the impact of CSR schools on student achievement in contrast to a carefully matched sample of comparison schools in Arizona. We subsequently replicated this study with the 2002 CSR schools and extended it to affluent school comparisons (McCaslin et al., 2005). An important extension of this work by Burross (2008) appears later in this issue.

The delay also allowed Mary McCaslin, the project’s codirector, to spend a year in one school observing and talking with the principal, staff, and students. This experience was invaluable for developing new instruments sensitive to students’ languages, beliefs, and feelings in Grade 3–5 classrooms.

THE SAMPLE

Principals in 21 of the 23 schools agreed to participate and were interviewed at their schools. The demographics of the two declining schools were represented in our sample; thus, we believe our sample represents well those schools awarded CSR funds in Arizona. Participating schools served students of poverty concentrations that ranged from 41% to 100% (M = 73%) and students who varied in ethnicity (although the dominant ethnicity across all schools is Hispanic). Schools also widely varied in size (range of 43–1,131 students; M = approximately 600; most schools served between 400 and 700 students). Schools were located in small cities and rural and urban areas. Teacher and student populations changed from year to year, sometimes markedly; thus, our data are necessarily a snapshot of evolving schools. More information about individual schools can be found in Table 1.

We selected Grades 3–5 as our primary focus to study school reform because we were concerned that investment in Grade 3 as the benchmark of successful school reform was misplaced. We suspected that third grade may well be the peak of student-motivated learning and thus underrepresents the difficulty of helping students pursue and maintain learning progress. We were especially concerned about the changes associated with student learning and motivation in Grades 3–5—which, as noted earlier, we call the “fourth-grade window”—to better capture how students mediate classroom practices and how student motivation might inform identity and achievement.

WHAT DID WE STUDY?

The articles that follow describe what we studied, how we measured it, and what we found. In the next article, Tom Good (2008) describes CSR from the principals’ perspective, including their beliefs about school reform, the students they serve, and the problems that teachers face in raising student achievement. In the third article, Caroline Wiley, Tom Good, and Mary McCaslin (2008) report what we observed in CSR classrooms over the course of one school year. These data are based on an observation rubric that captured instructional opportunities, student activity, and teacher-student relationships.

Fourth, Amanda Bozack, Ruby Vega, Mary McCaslin, and Tom Good (2008) continue the depiction of classroom processes with an analysis of observers’ narrative accounts that elaborated on the standard observation system. Specifically, this analysis addresses teacher support of students and student opportunities for choice and autonomy. In sum, the next three articles concern what principals think is happening in their schools and our observations of classroom practices.

Even though teachers were largely unaware of the timing of our visits, we still were faced with the possibility that our observations did not capture “standard” classroom practices. This seemed an especially important alternative hypothesis for us to consider. Our solution was to ask students about their views of their schools. In five case study schools (which represented a range of state-based school performance labels at the time of selection), students completed surveys about their school, learning, and sense of self, and wrote stories in response to pictures of routine classroom events. In the fifth article, Mary McCaslin (2008) explains how her participant observation project informed the instrumentation for capturing student perceptions of their school and classroom learning.

In the sixth article, Alyson Lavigne Dolan and Mary McCaslin present an analysis of student stories in response to a picture of student-teacher interaction. Pictures were meant to trigger any concerns that students had about their learning or their teacher. Girls responded to a picture of a female teacher returning to a female student a paper that looks to be a language arts-type assignment. Boys responded to a picture of a male teacher gesturing at a math problem on the board to a boy standing in front of the classroom.

Seventh, Ida Rose Florez and Mary McCaslin (2008) describe student stories in response to a picture of peers working in a mixed-gender small group. In the picture, one boy and one girl are sitting side by side facing the camera, across the table from them is one girl whose face is less visible, and other students in other groups are suggested. The group task is ambiguous, and there are no obvious signs of difficulties in group dynamics.

Research on student perceptions continues in the eighth article, by Mary McCaslin and Heidi Legg Burross (2008). They present individual differences in the motivational dynamics of students’ understanding of and disposition toward school.

Next, we turn to the basic question that drives school reform: Does school reform improve student performance on mandated tests? Heidi Legg Burross (2008) describes student performance data across 8 years of testing on our CSR school sample as compared with demographically matched schools and those that serve students of relative affluence.

Finally, Tom Good and Mary McCaslin (2008) discuss what we have learned from our project that can inform school reform matters. Here we assert an advanced organizer: School reform matters. One of our primary goals is to find ways to improve the likelihood of successful school reform for the betterment of students and families, and teachers and principals. School reform is needed, and it needs to be successful. One lesson from the 1980s is how readily society can turn its back on failed reform initiatives and the problems that they did not solve (McCaslin & DiMarino-Linnen, 2000). We cannot let this happen to schools and the students they serve. This article concludes with considerations derived from our findings that we believe can move school reform forward; strategies for doing this are provided for policy makers, principals, teachers, researchers, and teacher educators.

Acknowledgments

This research was supported by the U.S. Office of Educational Research and Improvement (OERI) Grant No. R306S000033. The authors take full responsibility for the work, and no endorsement from OERI should be assumed.

Notes

1. This brief review of literature includes information about school and teacher effects on student achievement up to 2002, the year we started data collection. In the concluding article in this volume (Good & McCaslin, 2008), we comment on the most recent literature.

2. This review summarized what was known or at least knowable about CSR when we started our research.

References

Averch, H., Carroll, S., Donaldson, T., Kiesling, H., & Pincus, J. (1974). How effective is schooling? A critical review of research. Santa Monica, CA: Rand.

Biddle, B. (1997). Foolishness, dangerous nonsense, and real correlates of state differences in achievement. Phi Delta Kappan, 79, 8–13.

Bloom H. (2002, May). Longitudinal analysis of the accelerated model. Paper presented at the Network of Researchers meeting, Washington, DC.

Brookover, W., Beadie, C., Flood, P., Schweitzer, J., & Wisenbaker, J. (1979). School social systems and student achievement: Schools can make a difference. New York: Praeger.

Borman, G. D., & D’Agostino, J. V. (1996). Title I and student achievement: A meta analysis of federal evaluation results. Educational Evaluation and Policy Analysis, 18, 309–326.

Borman, G., Hewes, G., Overman, L., & Brown, S. (2003). Comprehensive school reform and student achievement: A meta-analysis. Review of Educational Research, 73(2), 107–125.

Borman, G. D., Stringfield, S C., & Slavin, R. E. (Eds.). (2001). Title I: Compensatory education at the crossroads. Mahwah, NJ: Erlbaum.

Cohen, M. (1983). Instructional, management, and social conditions in effective schools. In A. Webb & L. Webb (Eds.), School finance and school improvement: Linkages in the 1980s (pp. 17–50). Cambridge, MA: Ballinger.

Coleman, J., Campbell, E., Hobson, C., McPartland, J., Meade, A., Weinfeld, F., et al. (1966). Equality of educational opportunity. Washington, DC: U.S. Department of Health, Education and Welfare.

D’Agostino, J. V., Borman, G. D., Hedges, L. V., & Wong, K. K. (1998). Longitudinal achievement and Chapter 1 coordination in high-poverty schools: A multi-level analysis of the Prospects data. Journal of Education for Students Placed at Risk, 3, 401–420.

Federal Interagency Forum on Child and Family Statistics. (1999). America's children: Key national indicators of well-being (NCES 1999-019). Washington, DC: Government Printing Office.

Good, T. L., Biddle, B., & Brophy, J. E. (1975). Teachers make a difference. New York: Holt, Rinehart and Winston.

Good, T., & Brophy, J. (1986). School effects. In M. Wittrock (Ed.), Handbook of Research on Teaching (3rd ed., pp. 570–604). New York: Macmillan.

Good, T. L., Burross, H. L., & McCaslin (2005). Comprehensive school reform: A longitudinal study of school improvement in one state. Teachers College Record, 107, 2205–2226.

Hanushek, E. (1997). Assessing the effects of school resources on student performance: An update. Educational Evaluation and Policy Analysis, 19, 141–164.

Jencks, C., Smith, M., Acland, H., & Bane, M. J. (1972). Inequality: A reassessment of the effect of family and schooling in America. New York: Basic Books.

Jensen, A. R. (1973). Educability and Group Differences. New York: Harper and Row.

Klaus, P. E. (1973). Yesterday's children: A longitudinal study of children from kindergarten into the adult years. New York: Wiley.

Klitgaard, R., & Hall, G. (1974). Are there unusually effective schools? Journal of Human Resources, 74, 90–106.

Ladd, H., & Hansen, J. (Eds.). (1999). Making money matter: Financing America's schools. Washington, DC: National Academy Press.

McCaslin, M., Burross, H. L., & Good, T. L. (2005). Change and continuity in student achievement from Grades 3 to 5: A policy dilemma. Education Policy Analysis Archives, 13(1), 1–18.

McCaslin, M., & DiMarino-Linnen, E. (2000). Motivation and learning in school: Societal contexts, psychological constructs, and educational practices. In T. Good (Ed.), Schooling in America: Yesterday, today, and tomorrow; 100th Yearbook of the National Society for the Study of Education (pp. 84–151). Chicago: University of Chicago Press.

McCaslin, M., & Good, T. (1996a). Listening in classrooms. New York: Harper Collins Press.

McCaslin, M., & Good, T. (1996b). The informal curriculum. In D. Berliner & R. Calfee (Eds.), Handbook of educational psychology (pp. 622–673). New York: Macmillan.

McCormack-Larkin, M., & Kritek, W. J. (1983). Milwaukee's Project RISE. Educational Leadership, 40(3), 16–21.

National School Boards Association. (1999, November 29). Ten critical threats to America's children: Warning signs for the next millennium. A report to the nation presented by the National League of Cities, National School Boards Association, Joe DiMaggio Children's Hospital, Youth Crime Watch of America. Retrieved April 25, 2008, from http://www.nsba.org/bookreports/ten_threats.pdf

Pogrow, S. (1999). Overcoming the cognitive wall: Accelerating the learning of Title I students after the third grade. In G. Orfield & E. Debray (Eds.), Hard work for good schools: Facts not fads in Title I reform. The Civil Rights Project, Harvard University, Cambridge, MA (mimeo). (ERIC Document Reproduction Service No. ED447247)

Rosenholtz, S. (1989). Teachers’ workplace: The social organization of schools. New York: Longman.

Ross, S. M., Alberg, M., & Nunnery, J. (1998). Selection and evaluation of locally developed versus externally developed schoolwide projects. Memphis, TN: University of Memphis.

Rowan, B., Bossert, S., & Dwyer, D. (1983). Research on effective schools: A cautionary note. Educational Researcher, 12(4), 24–31.

Rutter, M., Maughan, B., Mortimore, P., Ouston, J., & Smith, A. (1979). Fifteen thousand hours: Secondary schools and their effects on children. Cambridge, MA: Harvard University Press.

Slavin, R. (1999). How Title I can become the engine of reform in America’s schools. In G. Orfield & E. Debray (Eds.), Hard work for good schools: Facts not fads in Title I reform (pp. 86–101). The Civil Rights Project, Harvard University, Cambridge, MA (mimeo). (ERIC Document Reproduction Service No. ED447247)

Stiefel, L., Rubenstein, R., & Berne, R. (1998). Intra-district equity in four large cities: Data, methods, and results. Education Finance, 23, 447–467.

Stringfield, S., Millsap, M. A., Herman, R., Yoder, N., Brigham, N., Nesselrodt, P., et al. (with Gamse, B., Puma, M., Rosenblum, S., Beaumont, J., Randall, B., & Smith, L.). (1997). Urban and suburban/rural special strategies for educating disadvantaged children. Final report. Washington, DC: U.S. Department

of Education.

Teddlie, C., & Stringfield, S. (1993). Schools make a difference: Lessons learned from a 10-year study of school effects. New York: Teachers College Press.

Weber, G. (1971). Inner-city children can be taught to read: Four successful schools. Washington, DC: Council for Basic Education.

Wong, K. K., & Meyer, S. J. (1998). Title I schoolwide programs: A synthesis of findings from recent evaluation. Educational Evaluation and Policy Analysis, 20, 115–136.