Interpreting the Evidence on School Effectiveness
by Richard J. Murnane - 1981
A critical review of research on school effectiveness is presented. The research findings are applied in a discussion of the issues of quality of education, resource allocation, and public policies. (Source: ERIC)
The purpose of this article is to examine what has been learned from quantitative studies of school effectiveness and to assess the implications of the research results for public policy. Now is a particularly appropriate time to discuss this research because in these days of declining enrollments, severe budget constraints, and court-mandated school finance reform, the results of studies in this tradition are often cited in public policy debates concerning the role of public schools. These public policy debates frequently center on questions such as:
Are there systematic differences in the quality of education provided in public schools?
What school resources really make a difference?
What public policies should be implemented to improve the quality of education provided to disadvantaged children?
One of the goals of this article is to explain the contribution that research in this tradition has made in providing answers to these questions and to clarify what this research can and cannot tell us.
Section I presents a critical review of the results of quantitative studies of school effectiveness. Section II explains the limits of this type of research. In particular, this section points out why such research cannot provide reliable information about the effects on student achievement of policies designed to improve the school resources available to children. The crux of the message is that this type of research does not provide information about the behavioral responses of teachers, students, and families to changes in resource allocation mechanisms. Section III discusses strategies for taking into account the behavioral responses of the key actors in the educational process in formulating public policy.
I. QUANTITATIVE STUDIES OF SCHOOL EFFECTIVENESS
In the last fifteen years, a large number of quantitative studies of the relationship between school resources and student achievement have been conducted. Some are called input-output studies, others, educational production function studies, and others, simply multivariate studies of school effectiveness. Definitions of school resources have differed, as have the measures of student achievement. Despite these differences, these studies, which we shall call simply quantitative studies of school effectiveness, share a basic methodology and can be viewed as examples of a particular research approach. In this approach, no attempt is made to manipulate experimentally the school resources that children receive. Instead, it is natural experiments- the variation in school resources created by the operation of a school system-that provide the data base for analysis. In essence, the research strategy can be viewed as taking a snapshot of a school system at work. The key parts of the snapshot are information on the school resources that children receive at a point in time and one or more measures of student progress. Sometimes the snapshot also includes information about students family backgrounds. Multiple regression techniques are used to estimate the impact of individual school resources on student achievement.
In the last fifteen years we have learned a great deal about how to take more accurate snapshots of schools at work. In particular, we have learned the importance of using the individual child as the unit of observation, of using childrens progress as the measure of school effectiveness (instead of the students achievement level), and of identifying the school resources that each child actually receives (rather than using the average resources present in the school or the school district). In addition, the definition of school resources has become much broader and more sophisticated. The first studies focused on physical facilities, library books, student-teacher ratios, and school size. In recent studies, the definition of resources has been expanded to include characteristics of teachers and classmates, indicators of teacher quality, the amount of time devoted to learning tasks, and descriptions of instructional techniques. These improvements in methodology have increased the ability of research in this tradition to provide reliable information about the impact of school resources on student achievement in the particular times and places that are studied.
What have we learned from quantitative studies of school effectiveness? The most notable finding is that there are significant differences in the amount of learning taking place in different schools and in different classrooms within the same school, even among inner city schools, and even after taking into account the skills and backgrounds that children bring to school. The importance of this result, found in all four studies that have addressed this question, cannot be underestimated.* It provides clear support for the belief of most Americans -that schools matter. It also provides support for the position that it is worthwhile devoting attention to the question of why some schools provide better education than other schools do, despite our limited success in answering this question.
Having determined that more learning takes place in some schools and classrooms than in others, researchers turned to the question of whether the differences can be explained by differences in school resources. There is no unequivocal consensus regarding the role of any school resource in contributing to student achievement. However, a judicious interpretation of the evidence (including the research methodology as well as the pattern of coefficient estimates) does suggest some tentative conclusions. Before turning to discussions of individual resources, it is important to note that in all of the studies discussed in this essay, student achievement is measured by improvements in scores on standardized tests of cognitive skills. These tests are by no means problem free.3 However, they do provide the best available measures of student achievement that can be used in large-scale studies.4
To most Americans, quality of education is synonymous with quality of teaching. Thus, it is not surprising that the role of the teacher has been a central focus of quantitative research on school effectiveness. The research strategy used to study teachers has been to include measurements of teacher characteristics in the vector of school resources that is related to student achievement. The choice of the teacher characteristics included in any study has depended primarily on the availability of data. Thus, it is often difficult to compare results across studies. Despite this problem, however, the results have been informative.
Virtually every study of school effectiveness finds that some attributes of teachers are significantly related to student achievement, and certain attributes play an important role in several studies. In particular, the intellectual skills of a teacher, as measured by a verbal ability test,5 or the quality of the college the teacher attended6 tend to be significant. Teachers with some experience are more effective than teachers with no experience7although one study reports a significant exception to this conclusion.* Teachers with high expectations for their students are effective in helping children to acquire cognitive skills.9 Recent studies in which large samples were examined indicate that there are significant interaction effects between the characteristics of teachers and students.10 In other words, some teachers are more effective with certain types of students than with other types of students.
One interesting negative result present in many studies is that teachers with masters degrees are no more effective on average than teachers with only bachelors degrees. At the same time, studies have found that teachers who voluntarily attended postgraduate courses are particularly effective. This suggests that voluntary participation in postgraduate education may be a signal of high motivation-an attribute that is difficult to measure, but that administrators think is crucial to a teachers effectiveness. It may be that when the pay increment for possession of a masters degree was first introduced into teachers salary schedules, it was justified by productivity differences. At that time, only a small percentage of teachers had masters degrees, and these may well have been the most highly motivated teachers. Today, however, when a majority of teachers have advanced degrees, and when some states require that teachers obtain MAs to earn permanent positions, the degree is no longer a signal of a particularly high level of motivation.
One final result concerning teachers is that supervisors know in general who the more effective teachers are. Two studies12 have analyzed the relationship between principals evaluations of teachers and the effectiveness of the teachers as measured by their students progress on standardized tests. In both studies, the evaluations were significantly related to student test score gains (and there is evidence that the evaluations were not based on the test results).
The school-related research on peer groups asks whether a childs achievement (or attitudes) is affected by the characteristics of the children with whom he or she interacts in school. This is an extremely important public policy question since peer groups are a resource that cannot be equalized by simply providing more dollars to schools serving needy children. If peer groups are critical, as Coleman suggested in his 1966 report, the meaning of equality of opportunity must be reconceived.
Two problems have hindered research on peer-group effects in schools. The first problem is the difficulty in identifying the peer group. In practice, the characteristics of individual data bases determine whether a childs peer group is defined as the other children in the classroom, in the grade level, or in the school as a whole. Whether a particular definition provides accurate information about the children with whom a child actually interactsdepends on the organization of the school-in particular, on the extent to which selfcontained classrooms, tracking, and homogeneous grouping are used. Only rarely have studies even attempted to control for grouping practices.
The second problem concerns the attributes of peers. Most parents want their children to interact with other children who share their values and are motivated to succeed in school. However, these noncognitive traits are very difficult to measure. As a result, in most studies peers are characterized by race, achievement, or family income. Differences in results across studies may be due to the fact that in different samples, the observed characteristics of peers are differentially related to the unobserved values and attitudes. The significance of this problem for public policy is discussed in Section II.
Despite these problems, peer-group research has begun to reveal some patterns. In particular, there is evidence that elementary school children with low initial skill levels who attend schools in which the average achievement level is relatively high make more progress than such children who attend schools in which the average achievement level is relatively low.13 There is similar evidence regarding socioeconomic status (SES). Elementary school children from low SES families who attend schools with a high proportion of high SES students make more progress than children who attend schools in which most children come from low SES families.14
The evidence in regard to racial composition is more difficult to interpret. Some evidence suggests that both black and white students who attend schools in which the racial composition is in the 40-60 percent range make more progress than students in schools that are more segregated by race.15 Other evidence suggests that racial composition does not matter to either white or black students until the proportion of black students becomes quite high. Above a critical level (perhaps different for black and white students), achievement is decreased as the proportion of black students increases.16 Still other evidence indicates that black students who once attended racially segregated elementary schools subsequently do less well in racially mixed junior high schools than in segregated schools.17
The explanation for the differences in the racial composition findings may lie in the extent to which racially desegregated schools were in fact integrated in the sense that students felt comfortable and communicated with each other. For example, black children who moved from a segregated elementary school to a racially mixed junior high school may have encountered a desegregated but not truly integrated environment. The unfamiliar confrontation with many better prepared white students may have been a threatening, discouraging experience that led to lower achievement.
A final peer-group issue concerns the effect of student-body composition on the achievement of advantaged children. Summers and Wolfe found that the progress of children with high initial test scores was not subsequently affected by the ability distribution of the children in their schools.18 Henderson et al. found that children with high initial test scores gained just as much from being in classes in which the average achievement level was high as children with low initial test scores did.19 However, the effect on individual achievement of improvements in average class achievement was greater at the low end of the average achievement distribution than at the high end. The authors interpret this result as indicating that a policy of redistributing students in order to equalize the average achievement in every class would lead to large increases in the achievement of children in slow classes and small decreases in the achievement of children in fast classes.20
Thus, it appears that children disadvantaged by low initial achievement or low SES benefited from attending schools with more fortunate students, while the cost to the more fortunate students in these schools in terms of decreased achievement was small. As we discuss later, however, the definition of a small cost lies in the eye of the beholder; if parents feel that mixing the race, ability, or SES of students reduces the quality of education for their child, they may respond in a manner that defeats the policy.
The impact of class size on student achievement is perhaps the most thoroughly researched question in education. The reason is that class size is a highly visible indicator of quality to many parents and teachers; it is also a good indicator of per-pupil instructional costs since teachers salaries comprise the bulk of instructional expenditures. Consequently, the class size issue is of great interest to both advocates of better education and proponents of tax relief. Despite the extraordinary volume of research, there is no consensus on the role of class size. Evidence exists to support both smaller classes and smaller budgets. A recent synthesis of past research by Glass and Smith found that average student achievement was much higher in very small classes than in classes with twenty or more students.21 However, average achievement in classes with twenty students was only marginally higher than average achievement in classes with thirty or forty students. This does not offer much consolation to educators in urban areas concerned with increases from 28 students per class to 30 students.
Why is the role of class size so elusive? There are two parts to the answer to this question; both concern limitations in the ability of existing research to capture salient aspects of the education process. The first problem is that the effect of class size surely depends on a teachers instructional strategy. Class size would matter less in a class in which the teacher provided instruction to the entire class simultaneously than in a class in which the teacher relied heavily on individualized instruction. In principle, this interaction effect between class size and instructional strategy can be investigated using multiple regression if the sample size is sufficiently large. In practice, however, this is very difficult to do because reliable information on instructional strategy can be obtained only by using expensive observational techniques. As a result, studies using such techniques usually employ very small samples.
A second and related problem concerns the insensitivity of existing research strategy to the effects of class size on the children most affected by this variable. It seems plausible that the cost of a large class may not be borne proportionately by all of the students in the class. Instead, the cost is borne primarily by children with learning problems who do not profit from instruction geared to the average achievement level in the class. In a small class the teacher may be able to find the time to provide particular attention to such children. It is frequently not possible to examine this hypothesis effectively because children with special learning problems tend to be absent from school more often than other children.22 a result, they are very likely to miss at leastone of the two standardized tests that provide the measure of student progress. Consequently, children of this type have a disproportionately high probability of being excluded from samples used in school effectiveness studies.
Recently, attention has focused on classroom time as a school resource. Interest in the role of time stems from the fact that school policies concerned with the length of the school day, the school year, and the number of subjects that are studied all affect the amount of time available for work on basic skill development. The first results on the role of time are encouraging, in that several studies report systematic relationships between measures of time use and student learning. However, at this point it is difficult to interpret the results because the analyses have used three different definitions of time. The first definition is the amount of time children spend in school.23 The second is the amount of time devoted to basic skill development.24 The third is the amount of time children spend on task, actually working at basic skill development.25 Clearly, the third definition is the most relevant to learning basic skills. However, time on task is not a policy variable, and its relation to the definitions of time that can be manipulated by policy depends on the behaviors of students and teachers in ways that are not understood. The value
of research on the role of time in improving education will depend on the success of efforts to understand how teachers and students transform aspects of time that are subject to public policy into the amount of time students spend on task.
Physical facilities-for example, the number of library books in the school, the quality of the science labs, the size and age of the school-played a prominent role in early school effectiveness research. The reason for this interest was that physical facilities were the capital in the production process, and capital plays a central role in the economic models from which this research stemmed. However, the early studies did not find these indicators of capital to be systematically related to student achievement. (Moreover, as evidence began to accumulate concerning the importance of human capital, attention shifted to developing better measures of human resources.) Thus, the current conclusion is that the physical resources available in a school in a
particular year are not systematically related to the achievement of the students in that year.
Does this mean that physical facilities do not matter? Perhaps. However, an alternative interpretation is that the quality of the facilities influences which teachers and children attend a particular school. This mechanism is not captured in the snapshot methodology used in quantitative studies of school effectiveness. We will develop this argument in greater detail in Section II.
INSTRUCTIONAL STRATEGIES AND CURRICULUM
Instructional strategies and curriculum have long been the focus of a great deal of educational research. The primary reason is that research evidence indicating that particular instructional strategies or curricula were clearly better than alternatives would have direct implications for policy. Schools could purchase new curriculum packages. Colleges could train aspiring
teachers in the use of the most successful instructional techniques.
Unfortunately, despite a great many studies and countless publications, no unequivocally superior curricula or instructional strategies have been found. Many studies report that students achieved at an exceptionally rapid rate when taught with a particular curriculum or instructional strategy. However, time after time, these successes have not been replicated in other sites, or even maintained in the original sites over a long period of time.
The most compelling explanation for the inability to replicate successes is that the same curricula and instructional strategies are used in very different ways in different sites. For example, Chall, in her well-known book Learning to Read: The Great Debate, points out that even the basicdistinction between the phonics approach to reading and the sight reading approach is not clearcut when one observes their use in a number of classrooms.26 Similarly, Van Deusen Lukas reports enormous variation in the actual educational practices taking place in classrooms using the same innovative instructional approach.27
Developers of innovative curricula or instructional strategies often interpret these findings as evidence that the problem lies in the lack of fidelity to the technical characteristics of the particular curriculum or instructional technique. Implicit in this view is the assumption that teaching and learning can be viewed as a stable, well-defined production process, similar to growing hybrid corn. Fidelity to the details of the superior technology is thought to be possible and to result in increased productivity.
An alternative response to the evidence on the variation in practice is that such variation is unavoidable and in fact is crucial to effective teaching. A necessary condition for effective teaching may be that teachers adapt instructional strategies and curricula to their own skills and personalities, and to the skills, backgrounds, and personalities of their students. In this view of
teaching and learning, the technical characteristics of instructional strategies and curricula are not, by themselves, the critical components. Instead, what matters is the extent to which teachers are willing and able to adapt the curricula or instructional strategy to their needs and to the needs of their students.28
SUMMARY OF SCHOOL EFFECTIVENESS RESULTS:
PRIMARY RESOURCES AND SECONDARY RESOURCES
We have learned a great deal from quantitative research on the determinants of school effectiveness. The most important lesson is that schools make a difference. Even in inner cities in which virtually all of the children attending public schools come from relatively poor families, there are important differences in the amount of student learning taking place in different schools and even among classrooms in the same school. A second lesson is that teachers are a critical resource. Children learn more when they are taught by talented, highly motivated teachers who believe that their students can learn and who structure the school day so that students spend large amounts of time on task, working at basic skill development. We have learned a little about how to identify such teachers. However, it also appears that no set of observable characteristics provides a reliable composite picture of the effective teacher.
The research results also indicate that the composition of the student body matters. In the natural experiments that have been studied, disadvantaged children who attended schools that served a significant number of children from more advantaged backgrounds profited from this experience.
Quantitative research on school effectiveness began with a broadly specified input-output model that was agnostic on the role played by particular school resources. In the model, a large number of resources were treated in parallel fashion. A critical survey of this research indicates that the primary resources are teachers and students. It is on these human resources that researchers should concentrate, since they are poorly understood, play a central role in policy choices, and appear to dominate other resources.29
Physical facilities, class size, curricula, and instructional strategies can be seen as secondary resources that affect student learning through their influence on the behavior of teachers and students. This perspective has two significant implications. First, current research methodology, which employs a snapshot approach to examine the impact of school resources on student achievement, may be inappropriate for measuring the influence of secondary resources. For example, it may be that these resources affect student achievement by influencing which teachers and children are found in particular schools. This mechanism, which is described more fully in Section II, is not captured with the snapshot methodology. The second, and related, implication is that research on the role of these secondary resources should concentrate on their impact on the behavior of teachers and students. We will return to this theme in Section III.
II. THE POLICY PROBLEM
In a nutshell, the policy problem is how to design policies that will provide more children with the school resources that contribute to rapid learning. Part of the difficulty in fulfilling this task stems from our limited understanding of what these resources are. However, research results provide increasing guidance concerning the resource configurations that are associated with high rates of student learning in ongoing educational systems.
A greater difficulty stems from the fact that resource configurations in ongoing systems result from a large number of institutional mechanisms, internal labor-market rules and customs, and the responses of teachers and students and families to these mechanisms. For example, the allocation of teachers to schools is determined by seniority rules and the decisions of the more senior teachers. Which children attend particular schools is determined by rules concerning attendance boundaries, and by family location decisions. The relationships between resources and student achievement that are observed in the natural experiment research are conditional on the resource configurations present in the school system. The process that created these resource configurations is not considered in the analysis.
To change the resource configurations in a systematic way requires altering one or more of the formal or informal institutional mechanisms. Any alterations in the institutional mechanisms will elicit behavioral responses on the part of teachers or pupils and their families. These behavioral responses may well alter the very relationships just surveyed between observable inputs and student learning.
Some readers may believe that the preceding paragraph simply reflects the excuses of a timid researcher afraid to pursue the policy implications of his work. They may point out that in the substantive area from which this research tradition stems-production of hybrid corn-rules of thumb were also used by tradition-bound farmers in determining combinations of seed, fertilizer, and other inputs. Yet there is clear evidence that convincing these farmers to abandon their rules of thumb and instead to allocate resources in the proportions indicated by the research findings resulted in significant increases in their productivity. Why is education so different?
The key difference is that in corn production, the key inputs, seed, water, and fertilizer, are inanimate and their productivity depends only on the resource mix and on the weather, not on the method by which the resource allocation is determined. In education, the key resources are students and teachers, whose behavior and productivity are very sensitive to the methods used to allocate resources. This does not mean that policies cannot be altered. However, it does mean that effective policy analysis must take into account the behavioral responses that changes in resource allocation mechanisms will elicit.
Two examples may help to clarify the role of behavioral responses. The first concerns policies designed to take advantage of peer-group effects. Recall that research has indicated that low SES children who attend schools with more affluent peers make more academic progress than poor children who attend schools with uniformly poor students. This has led to a number of policies designed to increase the mixing of students by class, ability, or race. There has been enormous variation in the success of these policies. However, in a significant number of cases, the anticipated beneficial results have not been realized.
The reason may be that the middle-class children who attend integrated neighborhood schools voluntarily as a result of their parents decision to live in an integrated neighborhood may be different in unobserved critical ways from middle-class children who attend schools that are desegregated as a result of a conscious policy such as court-ordered busing. In particular, parents choosing to live in integrated neighborhoods and to send their children to public schools reveal by their choices the belief that public schools can provide their children with an adequate education. This belief, coupled with parental support and positive attitudes toward the other children in the school, may be critical in making the school a place where all children can learn. It may be for this reason that the research results indicate that poor children who attend such schools learn more than poor children who attend schools segregated by class or race.
Parents choosing to live in middle-class enclaves may not share these attitudes toward urban public education and toward children from poor families. Without these critical but unobserved attitudes, the policy of mixing children from different classes may not result in high-quality education.
The second example concerns declining enrollments and teacher layoffs. Many school districts, faced with declining student enrollments and severe fiscal constraints, are forced to lay off a significant number of teachers. In most districts the layoffs are determined by seniority rules. However, some administrators have argued that this is inefficient since under this system many effective teachers are laid off while less effective but more senior (and more expensive) teachers are retained. In some districts, administrators have dictated that those teachers who are designated by their principals as less effective will be laid off. Advocates of this policy point to the research evidence indicating that teachers do differ significantly in their effectiveness and that the evaluations of administrators do reflect teacher performance.
There is very little systematic evidence concerning how either layoff policy has affected the quality of education provided to children. However, there is limited evidence, much of it anecdotal, that the latter policy has been less successful in some districts than was hoped, for several reasons. First, effective teachers may resign, not because they anticipate losing their positions, but rather because they find that the competitiveness bred by this system diminishes the enjoyment that they derive from their job.30 Second, the quality of education provided in schools in these districts may decline as teachers adjust their behavior to take into account the fact that they are being compared with their colleagues. This can take the form of reluctance to share teaching materials or to help a fellow teacher deal with a particularly difficult child. Third, over time, as teachers alter their behavior, principals may find that their evaluations of teachers no longer reflect performance as well as they once did.31 (The studies that found that principals evaluations accurately reflect teacher performance were carried out in districts where this information was not used in layoff decisions; consequently, the evaluations did not evoke the behavioral responses just described.)
The point of these two examples is to illustrate the types of behavioral responses that policies designed to alter resource allocations can elicit. In some cases the behavioral responses are obvious-for example, when middleclass families withdraw their children from public schools rather than have them participate in a busing program. In other cases, the results may be more subtle. For example, in terms of socioeconomic status and other observable indicators, parents whose children are bused to desegregated schools may appear identical to those of children living in urban areas and attending neighborhood schools with many poor children. However, in unobserved dimensions, such as attitudes, the parents may be quite different, and the schools may be made different by contrasting levels of parental support.
The point of this section is not to argue that nothing can be done. There is a wide range of policies that can be used to alter resource allocations. Each of these will elicit a behavioral response, but the responses will differ. For example, the creation of magnet schools is an alternative to busing for promoting school desegregation. Unlike busing, magnet schools may evoke the positive parental support that is important to successful schooling. Early retirement programs are an alternative to layoffs for reducing the size of the teaching staff. These programs may permit the retention of talented young teachers without evoking the dysfunctional behavior that may accompany layoffs based on merit. The central point is that policy planning must take into account the behavioral responses that policies designed to alter resource allocations will elicit.
III. PRIMARY RESOURCES AND BEHAVIORAL RESPONSES:
NEW QUESTIONS FOR RESEARCHERS AND POLICYMAKERS
This article emphasizes the importance to student achievement of behavioral responses by teachers and students, the primary resources of schooling. These responses to institutional rules, and to the quantity and quality of secondary resources, determine first of all which children and teachers will participate in public schooling. They also influence the attitudes, expectations, and motivations of the participants and ultimately the quality of the learning environment in particular schools and classrooms. Given the importance of these behavioral responses, it seems important to learn about them. The following are a sample of research questions motivated by the behavioral response perspective:
What factors influence teachers participation in, and departure from, schooling as a career? In particular, under the existing system of compensation for teachers, which rewards longevity and degrees, are effective teachers more likely to leave public school teaching than ineffective teachers are?
Are particular working conditions critical determinants of teachers decisions to leave public school teaching?
Does class size influence the way teachers allocate classroom time among students? Under what circumstances do teachers change their instructional techniques in response to a significant change in class size?
What types of secondary resources (e.g., curricular alternatives, supplies and materials, preparation time) aid teachers in their search for instructional strategies that work for them and their students?
What types of programs or opportunities would induce middle-class parents to send their children to urban public schools?
How do different policies to curb violence in schools influence the behavior of students and, consequently, the learning environment?
In some respects, the research needed to answer these questions is very different from earlier research on the role of school resources in determining childrens achievement. The new research agenda focuses on the responses of human resources to incentives provided by institutional rules and to the opportunities and constraints provided by secondary resources. Earlier research treated all school resources as parallel; moreover, it reflected the assumption that resource configurations could be manipulated and packaged by officials. This new research agenda pays particular attention to the determinants of resource configurations. In other words, it explores the impact of institutional rules and the quality of secondary resources on the mobility decisions of teachers and families.
While these differences are significant ones, the new research agenda has grown directly out of the earlier research on school effectiveness. Clear evidence from earlier research that schools matter, plus the puzzles created by ambiguous findings on particular resources, led to the perspective developed in this article. In this respect, the research directions suggested here are a natural successor to the earlier snapshot research. In time, research on the behavioral responses of teachers, students, and families may enable us to choose public policies with a clear sense of their impact on school effectiveness. However, the research questions are extraordinarily difficult to answer. Consequently, it will be many years before researchers can provide policymakers with reliable predictions concerning the results of particular policy changes in school systems.
Given this situation, it seems important to ask whether there are alternatives to research for taking behavioral responses into account in the decision-making process. Lindblom has argued that the decision-making process itself can sometimes solve the problem of developing resource allocation mechanisms that evoke productive, rather than debilitating, behavioral responses.
32 A systematic exposition of this argument is beyond the scope of this article. However, a brief discussion of teachers unions and collective bargaining may illustrate the argument.
Effective union leaders know which dimensions of working conditions for example, class size, preparation periods, protection against violence-are most important to local teachers. They also know what types of resource allocation mechanisms-for example, merit pay-are disliked by their members. The process of collective bargaining reveals these preferences and provides information about their relative importance. When conducted by skilled negotiators in a framework that represents the interests of children and families as well as teachers, collective bargaining can produce resource allocation mechanisms that avoid debilitating behavioral responses.33
Unions may play a role, not only in articulating preferences, but also in influencing teachers behavioral responses to new institutional incentives. For example, many districts have introduced early retirement programs in recent years in the hope of inducing older teachers, especially those who are less effective, to retire, thereby reducing the necessity of laying off younger teachers. Some observers have doubted that these programs will succeed because older teachers may react with resentment, feeling that the early retirement choice is an admission that one can no longer function effectively in the classroom. Also, some teachers fear that the existence of an early retirement option could lead to pressure on older teachers to resign. Defensive reactions to this fear could have unexpected and undesirable consequences.
The union can play an important role in facilitating the success of early retirement programs by giving them legitimacy and guaranteeing their integrity. In other words, union support for early retirement programs can give them the status of legitimate benefits, earned through years of service, instead of a dole, distributed to burned-out teachers. Moreover, the existence of a well developed grievance procedure can ease fears that the early retirement program would lead to harassment of older teachers. Thus, the existence of a teachers union may be important in stimulating constructive responses to policies such as early retirement programs.
The point of the teachers union example is not to make a blanket rationalization for collective bargaining. It is one of many alternative forms of decision making. Other forms include voting and delegation to professionals. The decision-making form that will elicit the most productive behavioral responses will depend on the participants, the issue, and the setting.
The point we would like to emphasize is that choices about decision-making forms are extremely important. In our view of the production process for schooling, resources do matter. However, the relationships between the primary inputs-teachers, students, and families-and the outputs-student skills-depend critically on the behavior of the key actors. Their behavior is sensitive to the incentives provided by the school system. Unfortunately, the nature of the responses of these key actors to particular incentives is not well understood. In this view, interest groups such as teachers unions and parents associations can play a positive role by providing information about critical behavioral responses, and in some cases by influencing these responses. Viewed in this perspective, a key policy question is what form of decision making will be most successful in eliciting the critical information about behavioral responses. The effectiveness of public schooling depends to a large extent on our ability to develop and use such decision-making processes effectively.
WHAT HAVE WE LEARNED?
The purpose of this article is to provide an interpretation of school effectiveness research that explains puzzles in the empirical findings and provides a coherent perspective from which to ask new research and policy questions. At this point it may be helpful to recapitulate the basic themes developed in this discussion:
1. There is compelling evidence that schooling makes a difference in determining the cognitive skills of children. Consequently, the search for strategies to make schooling more effective is a worthwhile quest.
2. The primary resources that are consistently related to student achievement are teachers and other students. Other resources affect student achievement primarily through their impact on the attitudes and behaviors of teachers and students.
3. The central school resources-teachers and students-will respond to any changes in the institutional rules, customs, or contract provisions that determine the allocation of resources. Some of these behavioral responses will enhance student achievement; others will diminish achievement. The nature of the responses will depend on the priorities and opportunities of these key actors.
4. Better data and more research will help us to learn more about the relationships between school resources and student achievement in ongoing educational systems. However, quantitative research on school effectiveness, as currently conducted, will not provide reliable information about the effects of changes in resources on student achievement. The reason is that the methodology does not address the question of how resources are allocated in ongoing systems. Therefore, new approaches need to be developed and applied.
5. A central problem in improving schools is to develop mechanisms for incorporating into the decision-making process information about the priorities of the key actors, and consequently about their likely behavioral responses. The quality of public education in the future will be determined not only by the level of resources available, but also by our success in developing policy processes that take into account the behavioral responses of teachers, students, and families.
1 See E. A. Hanushek. Conceptual and Empirical Issues in the Estimation of Educational
Production Functions, The Journal of Human Resources, Summer 1979. for a detailed
description of the methodology used in school effectiveness research.
2 D. Armor et al.. Analysis of the School Preferred Reading Program in Selected Los Angeles
Minority Schools (Santa Monica, Calif.: The Rand Corporation. 1976): E. A. Hanushek,
Teacher Characteristics and Gains in Student Achievement: Estimation Using Micro Data.
The American Economic Review 61 (1971): 280-88: R. J. Murnane, The Impact of School
Resources on the Learning of Inner City Children (Cambridge: Ballinger. 1975): and R. J,
34 Teachers College Record
Murnane and B. R. Phillips. Effective Teachersof Inner City Children: Who They Are and What
They Do, 1979, mimeo.
3 For example. see H. A. Averch et al., How Effective Is Schooling? A Critical Review and
Synthesis of Research Findings (Santa Monica, Calif.: The Rand Corporation, 1972).
4 An alternative to norm-referenced tests is criterion-referenced test, which are more sensitive
to differences in curricula. However, to use such tests to compare curricula or school programs,
there must be agreement on the goals of the programs. J. R. Murphy and D. K. Cohen
(Accountability in Education-The Michigan Experience, The Public Interest, vol. 36, 1974)
document how difficult it is to reach agreement on this issue. The widespread interest in the
National Assessment of Educational Progress (NAEP) suggests that it is possible to develop
instruments that measure proficiency in a number of skills that are commonly regarded as
important. However, it is not coincidental that the design of the data collection in NAEP prevents
analysis of the effectiveness of particular educational programs.
5 Hanushek, Teacher Characteristics and Gains in Student Achievement: and idem,
Education and Race (Lexington, Mass.: D. C. Heath, 1972).
6 A. A. Summers and B. L. Wolfe, Do Schools Make a Difference, The American
Economic Review 67 (1977): and D. R. Winkler Educational Achievement and School Peer
Group Composition," The Journal of Human Resources 10 (1975): 189-204.
7 Hanushek. Education and Race; Murnane. The Impact of School Resources on the
Learning of Inner City Children; and Murnane and Phillips. Effective Teachers of Inner City
8 Summers and Wolfe (Do Schools Make a Difference) found teaching experience to be
negatively related to the achievement of children with low initial achievement. They suggest that
this may be due to the fact that the undampened enthusiasm of new teachers makes them
particularly effective with slow learners, while the skills developed through experience are
particularly important in teaching children with above-average achievement. This is certainly
plausible. However, these results could also be due to a particular type of selection mechanism.
Effective experienced teachers may be more likely than ineffective teachers to leave exhausting
positions in schools serving large numbers of low-achieving children because they face a more
attractive opportunity set, both inside and outside the teaching profession. This selection process
could explain the negative relationship between teaching experience and effectiveness in
teaching children with low initial achievement. Such selection processes are explained in more
detail later in the paper.
9 C. R. Link and E. C. Ratledge, Student Perceptions, IQ, and Achievement, The Journal
of Human Resources I4 (1979).
10 H. J, Kiesling et al., Educator Objectives and School Production, 1979. mimeo; and
Summers and Wolfe, Do Schools Make a Difference.
1 I Hanushek, Teacher Characteristics and Gains in Student Achievement.
12 Armor et al., Analysis of the Schools Preferred Reading Program in Selected Los Ange1es
Minority Schools; and Murnane, The Impact of School Resources on the Learning of Inner City
13 V. Henderson, P. Mieszkowski, and Y. Sauvageau. Peer Group Effects and Educational
Production Functions,Journal of Public Economics lO( 1978): 97-106; and Summersand Wolfe,
Do Schools Make a Difference.
14 Winkler, Educational Achievement and School Peer Croup Composition.
15 Summers and Wolfe, Do Schools Make a Difference.
I6 Hanushek, Education and Race.
17 Winkler, Educational Achievement and School Peer Group Composition.
18 Summers and Wolfe, Do Schools Make a Difference.
19 Henderson et al., Peer Group Effects and Educational Production Functions.
20 Hendersons interpretation of peer group results is cited in order to clarify the somewhat
complicated nature of these findings. In fact, however, the natural experiment evidence does not
School Effectiveness 35
provide reliable evidence concerning the effects of a conscious policy of redistributing students.
(See Peer Group Effects.) The reason is explained in Section II of this article.
21 G. V. Glass and M. L. Smith, Meta-analysis of Research on the Relationship of C/ass-size
and Achievement (San Francisco: Far West Laboratory for Educational Research and Development,
22 Murnane. The Impact of School Resources on the Learning of Inner City Children.
23 D. E. Wiley and A. Harnischfeger, Explosion of a Myth: Quantity of Schooling and
Exposure to Instruction, Major Educational Vehicles. Educational Researcher 3 (1974): 7-12.
24 Kiesling et al., Educator Objectives and School Production.
25 B. S. Bloom. Time and Learning, American Psychologist 29 (1974): 682-88; and J. A.
Thomas, Resource Allocation in Classrooms, final report to the National Institute of
26 J. Chall. Learning to Read: The Great Debate (New York: McGraw-Hill, 1967).
27 C. Van Deusen Lukas, Problems in Implementing Head Start Planned Variation
Models, in Planned Variation in Education, eds. A. Rivlin and P. M. Timpane (Washington,
D.C.: The Brookings Institution, 1975).
28 Paul Berman and Milbrey W. McLaughlin, Federal Programs Supporting Educational
Change, Vol. VIII: Implementing and Sustaining Innovations (Santa Monica, Calif.: The Rand
29 The research surveyed in this article focuses on resources available at the classroom level.
As a result. the role of school principals is not considered. It seems intuitive that principals should
also be considered among the primary school resources that affect student achievement.
30 P. W. Jackson, Life in Classrooms (New York: Holt, Rinehart & Winston, 1068), pp.
31 The problem of dysfunctional behavior created by attempts to base compensation on
perceived productivity is not unique to publiceducation. Severaleconomists haveargued that the
strict internal labor-market rules that govern resource allocation in many industries are a
response to the problems of measuring the productivity of individual workers. See L. Thurow.
Generating Inequality: Mechanisms of Distribution in the U.S. Economy (New York: Basic
Books, 1976); and O. E. Williamson, M. L. Wachter. and J. E. Harris, Understanding the
Employment Relation: The Analysisof Idiosyncratic Exchange, The Bell Journal of Economics
6 (1975): 250-78, for different versions of this argument.
32 C. E. Lindblom. The Science of Muddling Through, Public Administration Review, 19
33 R. B. Freeman and J. L. Medoff, The Two Faces of Unionism, The Public Interest 57
This special issue features articles concerning first-hand accounts of literacy campaigns in developing countries, reassessments of the relationship between education and economic growth, the role of the individual in social change, and the arts as nonformal education and includes an expanded book review section.
Fernando Cardenal Richard Katz
Robert Coles David Kerr
Joseph Featherstone Henry M . Levin
Paulo Freire Kenneth B. Liberman
Gerald W. Fry Marvin K. Mayers
Henry Louis Gates, Jr. Nellie Y. McKay
C.T. Hu Valerie Miller
Mokubung O. Nkomo
Abel Prieto Morales
Stephen L. Walter
Pamela Barnhouse Walters
Harvard Educational Review
Longfellow Hall, 13 Appian Way,
Cambridge, Massachusetts 02138
Please begin my subscription with the Special
One Year 0 $21.50 0 $30.00 Also, please send me copies of Educa-
Two Years 0 $35.00 0 $50.O0 tion as Transformation at $7.50 each. 20% dis-
(Outside U.S. add $4.00 per year for postage.) count on orders of 5 or more.
0 Payment enclosed
Charge to: 0 Master Card 0 Visa