How Do School Districts Affect the Quality of Professional Development Provided to Teachers? Results from a National Sample of Districts
by Laura M. Desimone, Andrew C. Porter, Beatrice F. Birman, Michael S. Garet & Kwang Suk Yoon - 2002
This study examines the policy mechanisms and processes that districts can use to provide high-quality in-service professional development for teachers. The findings are based on a national probability sample of district professional development coordinators in districts that received federal funding from the Eisenhower Professional Development Program. We found that certain management/implementation strategies, such as aligning professional development to standards and assessments, continuous improvement efforts, and teacher involvement in planning, are associated with the provision of higher quality professional development for teachers. Characteristics of high-quality professional development are taken from the research literature and include active learning opportunities, duration, collective participation (e.g., participation of teachers from the same department, grade level, or school), and type of activity (e.g., traditional workshop vs. a more reform-oriented approach, such as teacher networks or study groups.) Thus, in the context of an evaluation of the nationís largest investment in teachersí professional development, this study provides empirical support, from a national probability sample of Eisenhower district coordinators, for the link between federal policies and strategies of support, implementation, and the quality of teachersí professional development.
This study examines the policy mechanisms and processes that districts can use to provide high-quality in-service professional development for teachers. The findings are based on a national probability sample of district professional development coordinators in districts that received federal funding from the Eisenhower Professional Development Program. We found that certain management/implementation strategies, such as aligning professional development to standards and assessments, continuous improvement efforts, and teacher involvement in planning, are associated with the provision of higher quality professional development for teachers. Characteristics of high-quality professional development are taken from the research literature and include active learning opportunities, duration, collective participation (e.g., participation of teachers from the same department, grade level, or school), and type of activity (e.g., traditional workshop vs. a more reform-oriented approach, such as teacher networks or study groups.) Thus, in the context of an evaluation of the nation's largest investment in teachers' professional development, this study provides empirical support, from a national probability sample of Eisenhower district coordinators, for the link between federal policies and strategies of support, implementation, and the quality of teachers' professional development.
The professional development of teachers is a key component of national efforts to improve student learning through teaching to high standards. Because school districts have a prominent role in defining, sponsoring and providing professional development (Little, 1989, 1993; Little et al., 1987), teachers' experiences in professional development activities can be shaped in positive or negative ways, depending on district strategies. Using a national probability sample of district professional development coordinators in districts funded by the Eisenhower Professional Development Program, this study investigates (1) the extent to which Eisenhower districts nationwide are practicing particular management and implementation strategies and (2) the extent to which these district strategies are associated with the quality of the professional development activities that they offer to teachers.i
This district analysis is part of a larger national study of the Eisenhower Professional Development Program. Authorized as part of Title II of the Elementary and Secondary Education Act (ESEA), The Eisenhower Professional Development Program was the U.S. government's largest investment solely focused on developing the knowledge and skills of classroom teachers in mathematics and science.ii Appropriated at approximately $335 million in 1999, the program distributes funds to districts by formula through state education agencies (SEAs) and through state agencies for higher education to institutions of higher education and nonprofit organizations. The Eisenhower program is a source of funding for professional development activities, not a specific approach to professional development. Allowable activities are wide ranging and include workshops and conferences, study groups, professional networks and collaboratives, task force work, and peer coaching. Furthermore, Eisenhower funding does not exist in a vacuum. Eisenhower-assisted activities also may receive funding through states, school districts, and other federal programs. Therefore the information in this paper about the effects of district-sponsored Eisenhower-assisted activities is also applicable to professional development funded through other sources.iii
BACKGROUND: THE DISTRICT ROLE IN PROFESSIONAL DEVELOPMENT
Recent literature on school reform emphasizes the critical role of school districts in setting the context for professional development activities (Elmore & Burney, 1996; Knapp, Zucker, Adelman, & St. John, 1991; Spillane, 1996; Spillane & Jennings, 1997; Spillane & Thompson, 1997). A major challenge that districts face is making professional development activities high quality and consistent with the district and state reform goals (Massell, 1998). In the environment of state-driven standards-based reform efforts, districts can play a major facilitating role in communicating the state's instructional policy, lending coherence to it and influencing the type and nature of professional development opportunities available to teachers (Spillane, 1996).
Recent research focuses on the importance of school-based professional development that is embedded in the daily life of teachers (Corcoran, 1995; Little, 1982; Loucks-Horsley, Hewson, Love, & Stiles, 1998; U.S. Department of Education, 1999). However, even when professional development occurs at the school, districts often play a central role in its planning and implementation. School districts play a role in building a vision of school reform and designing professional development to support reform efforts (Elmore & Burney, 1996). District actions can influence how schools respond to state standards and assessments and determine to a large extent whether they have the capacity to respond (Massell, 1998). For example, districts may play a key role in conveying to administrators, teachers, and providers of professional development the implications of state and local standards and assessments for professional development activities, and how various sources of funding could be used to support these activities. Districts also may play a role in providing technical support to schools in tracking progress toward achieving professional development goals.
The importance of studying district professional development policies is amplified by two factors. First, professional development has a critical role iii mediating the success of standards-based reform efforts (Smith & O'Day, 1991). Second, although some teachers receive high-quality professional development that changes their practice in positive ways, most teachers' professional development consists of activities that do not have the features empirically shown to be related to increasing teacher knowledge and changing teaching practicelong duration with follow-up, participation of groups of teachers from the same school or department, a focus on content, active learning opportunities, and coherence (e.g. consistency with other goals and activities; Birman, Desimone, Caret, & Porter, 2000; Caret, Birman, Porter, Desimone, & Herman, 1999; Caret, Birman, Porter, Yoon, & Desimone, 2002).
RICH CASE-STUDY RESEARCH HIGHLIGHTS THE NEED FOR NATIONAL DATA
There have been several useful studies on the district role in implementing standards-based reform, and on district professional development in particular. For example, Goertz, Massell, and Chun (1998) examined 14 districts in five states to examine how they were aligning their systems with state policy and how state accountability systems were affecting the districts' strategies, including professional development; Spillane's (1996) study of two districts in Michigan provides detailed information about district responses, in professional development and other areas, to the expanding state role in policy making; and Elmore (1997) and Elmore and Burney (1999) studied one district in New York to examine how professional development was used to foster instructional improvement.
Although the literature on the district's role in professional development and standards-based reform in general provides rich, detailed data on how districts operate, nearly all of this literature is based on case studies of one or a few districts. In addition, descriptions or short-term evaluations of a particular program dominate studies of district professional development. Marsh (2000) reviewed the most recent literature on school districts and concluded that many of the studies focus on district response to a certain state policy, draw on a small number of districts that represent exemplary actors, and use qualitative case-study interviews with key stakeholders. These case studies provide valuable information about processes and interactions in districts, including analyses of within-district variation in policy implementation (e.g., Spillane, 1998). Although such information is essential to a deep understanding of the results of individual district action, we seek to extend and complement the case-study research with data from a national probability sample of districts with Eisenhower funding that provide information about large-scale patterns of district practice and the effects of such practice on the quality of professional development provided to teachers.
Our research questions were as follows: To what extent are districts nationwide engaging in effective management and implementation strategies? How do these strategies relate to the quality of professional development activities that districts provide? Although districts can have a prominent role in determining the quality of professional development activities that teachers experience, very little is known about national patterns of district strategies for management and implementation of professional development. We provide national* data on the extent to which Eisenhower districts engage in specific management and implementation strategies targeted toward their professional development program.
We then examine which district strategies foster high-quality professional development activities and the extent to which districts use effective strategies. Both researchers and practitioners recognize that the current system of professional development is "fragmented, ineffectual, and inefficient" (Corcoran, 1995, p. 8). Still, much of the professional development literature focuses on the optimal characteristics of individual professional development activities and not on organizational strategies for providing the activities. The work that does focus on organizational strategies is limited to case studies. In this study we use a national probability sample of Eisenhower district professional development coordinators to examine, on a national level, the prevalence of particular strategies and which strategies are successful in fostering the provision of high-quality professional development.
CONSTRUCTS DATA SOURCE, METHODS, AND MEASURES
IDENTIFYING MANAGEMENT AND IMPLEMENTATION STRATEGIES
The district management and implementation strategies that we examine are (1) the alignment of professional development activities with state and district standards and assessments; (2) coordination among multiple professional development programs; (3) "continuous improvement" efforts based on indicators, needs assessments, evaluation, and guidance; and (4) how districts involve teachers and other school staff in planning professional development efforts. We chose these key management and implementation strategies based on the literature on systemic reform (e.g., Smith & O'Day, 1991), which emphasizes the importance of establishing a vision of reform, aligning and coordinating all components of the policy system, including standards and assessments and other reform efforts and including participation from all levels of the policy system.
These management and implementation strategies also reflect the provisions of the Eisenhower Professional Development Program, and it is in the context of an evaluation of this program that these analyses were conducted (see section on data sources). The provisions of the program were grounded in the systemic reform literature and thus are consistent with the basic tenets of systemic reform.
BUILDING A VISION: ALIGNMENT AND COORDINATION
Articulating and establishing a reform vision can provide a framework for implementing and monitoring all aspects of reform (O'Day, Goertz, & Floden, 1995). For example, in a study of 12 districts in California, Murphy and Hallinger (1988) found that effective districts had superintendents with a strong vision of reform, which they expected all teachers to emphasize, and had professional development activities focused on districts priorities as they related to this vision.
State and district standards and assessments provide a vehicle for unifying reforms and professional development. Thus, one method of designing and developing a program of professional development is to align the activities, pedagogy, and curriculum with standards and assessments adopted by the state or district and to coordinate funding with other programs in the state and district to develop a coherent professional development reform strategy. Taken together, alignment and coordination can be thought of as indicators of the extent to which a district is implementing a coherent vision for professional development.
Alignment is a difficult concept to define and measure. Ideally, all aspects of an educational system are aligned with each other to support student learning to high standards. This might mean that textbooks, other materials, and instructional approaches that teachers use match state and local standards and assessments, that the professional development teachers receive helps them to use those materials and approaches appropriately, and that only what is valued is included in the curriculum or in assessments.
Evidence of alignment can be sought in many places. Webb (1998) notes that one could look for alignment in the consistency of content focus between professional development activities and standards and assessments; the extent to which professional development activities and standards and assessments are rooted in a common view of how students learn; and the extent to which professional development and standards and assessments reflect the view that all students must learn to high standards, which indicates attention to the learning needs of diverse students.
The importance of alignment is emphasized in the literature. Aligning professional development with standards and assessments could contribute to a coherent professional development program that is focused on goals and objectives that are important for student learning, and that support high standards for learning (Smith & O'Day, 1991; Cohen, 1995; Cohen & Hill, 2000, 2001). In addition, aligning professional development with standards and assessments can be one way that districts work to send a common message to teachers about appropriate instruction (Massell, 1998; Webb, 1998). Lessons from systemic reform indicate that fragmentation within the system decreases motivation for working on reform (Fullan, 1993, 1996) and that successful reform is in part contingent on establishing long-term goals and being able to articulate a clear vision (Cohen & Spillane, 1992). When policies, including professional development strategies, and reforms are unaligned, teachers recognize the inconsistency (Grant, Peterson, & Shojgreen-Downer, 1996); alternatively, when policies and reforms are aligned, this can work to encourage changes in instruction (Spillane & Jennings, 1997).
Elmore and Burney (1996) emphasize the critical importance of establishing a focus for guiding professional development activities in the district. In their seminal case study of professional development in New York City's Community School District 2, they describe a district that systematically identified instructional goals and objectives and designed a strategy for professional development to address directly those goals and objectives. The district's emphasis on instructional improvement focused its professional development activities. Elmore and Burney (1999) state that District 2's success was due largely to the integration of professional development in the district's reform strategy. They view this district as an "existence proof" that districts can "be agents of serious instructional improvement" and that districts can use professional development as a tool for the reform of schools. Other researchers also have emphasized the importance of establishing a district vision for professional development (Spillane, 1996), making professional development an integrated part of district reforms (Sparks & Hirsh, 2000), and using standards and assessments in shaping professional development activities (Cohen & Hill, 1998).
Besides alignment, another way for districts to focus professional development activities is through the coordination of multiple sources of funding for professional development activities. Cofundingthe contribution of funds from two or more programs to support the same professional development activityis a strong indicator of coordination among programs and an important measure of the integration of reform efforts in a district.
Districts often must balance many sources of funding for professional development. When districts have established a focus for their professional development, they may deploy these multiple sources of funding toward the same ends. Leveraging resources can have a role in affecting the quality of professional development (Corcoran, 1995). For example, Elmore and Burney (1996) describe the "multi-pocket budgeting" that occurred in their case district, where the district administration used funding from multiple sources to support its coherent professional development strategy. Cofunding activities with those of other federal and district programs can signal district attempts to build a coherent vision of professional development or to establish a unifying focus for potentially disparate professional development activities (e.g., Guskey, 1997).
In the literature, case-study districts do not generally provide a common vision to guide the use of professional development funds from a variety of sources or link professional development to other education reform efforts. For example, in a 2-year study of 22 districts in eight states, Massell (1998)' found that professional development resources were often fragmented into different funding streams. In addition, district-sponsored professional development has been criticized for being uncoordinated among other programs and offices in the district and not integrated with other policies and reforms (Schlechty & Whitford, 1983). Case-study research also indicates that districts generally are unaccustomed to planning portfolios of professional development activities strategically to achieve instructional goals (Elmore, 1993), and although district commitment to in-service education is important in ensuring teachers have adequate skills and knowledge, case-study districts often do not have a definitive strategy for accomplishing this (Floden et al., 1988). The research reported here identifies the extent to which these case-study findings generalize to a larger sample of districts.
IMPLEMENTING THE VISION: STRATEGIES FOR THE CONTINUOUS IMPROVEMENT OF PROFESSIONAL DEVELOPMENT ACTIVITIES
Evaluation and accountability mechanisms, when consistent with reform goals, can focus attention on attainment of goals and provide useful information on weaknesses that need to be addressed (O'Day et al., 1995).Performance indicators can provide target goals and benchmarks for measuring progress in the provision of high-quality professional development (Fuhrman, Clune, & Elmore, 1988). For performance indicators to have their intended effect, school districts must engage in a process of decision making that is grounded in data. Information about teachers' needs should assist in setting the goals and objectives for professional development. Evaluation data should provide one means for determining whether professional development activities are moving teachers toward these goals. These continuous improvement mechanisms serve as a means of implementing the vision of professional development, established through alignment, coordinated funding, and teacher involvement.
Districts can use indicators (e.g., school-wide data) strategically, to shape and target local priorities (Fuhrman et al., 1988). Loucks-Horsley et al. (1998) see three roles for indicators to play in designing and implementing a program of professional development. First, strategic planning that includes developing goals and objectives and delineating how progress toward achieving these goals will be measured is key to a successful professional development program. Second, conducting a needs assessment of teachers serves as an important step in planning a professional development strategy that accurately addresses the strengths and weaknesses of teachers in a particular district. Third, the quality of professional development is believed to increase when teachers and providers are held accountable for outcomes of professional development and when professional development is evaluated based on teacher and student outcomes (see also Guskey, 1997).
However, to have an effect on the design of professional development activities, continuous improvement means more than establishing goals, measuring progress toward these goals, assessing the needs of teachers and evaluating professional development activities. Continuous improvement also means communicating with schools and teachers about district goals; standards and assessments; and needs assessments and. evaluation data. This type of information could help schools integrate their activities with district goals and standards (Newmann, King, & Rigdon, 1997). The term continuous improvement implies a feedback loop, in which data about progress are part of continuous communication and where data become part of a discussion about strengths and weaknesses and future strategies and decisions.
PLANNING PROFESSIONAL DEVELOPMENT: DISTRICT, SCHOOL, AND TEACHER INVOLVEMENT
Planning for professional development activities can occur at any level of the formal school district hierarchy and can involve teachers and other school staff in a variety of roles. For example, administrators can plan professional development activities at the district level, with the advice of teachers or information from teachers about their needs. Alternatively, professional development can be planned at the school level, by full faculties, by teacher representatives, or by principals without the involvement of teachers. Additionally, in school districts, decisions about professional development may be made at some level in between the district and the school, by clusters of schools (e.g., all high schools or all elementary schools in the district). At whatever level, the goal of planning for professional development is to design activities and experiences that improve the quality of teaching and learning by supporting the needs of teachers.
Whether the planning for professional development occurs at the district, cluster, or school levels, the odds of meeting teacher needs are increased if teachers are involved in the planning. Experts agree that teacher involvement in planning contributes to high-quality professional development (Clark, 1992). Teacher involvement in planning can help ensure that professional development addresses the skills that teachers need and that they employ the learning strategies they find most useful (Clark, 1992; Loucks-Horsley et al., 1998); and teachers' formal involvement in planning professional development, for example on committees, often represents more responsibility and potential influence for the teacher than needs assessments or informal involvement. Further, teacher participation in decision making in general contributes to building professional community and shared goals (Newmann & Associates, 1996; Rosenholtz, 1991).
Participation of teachers in planning professional development at the school level appears to be especially important. Much of the current research base on school reform emphasizes the importance of approaches that involve improving whole schools (e.g., Bodilly, Keltner, Purnell, Reichardt, & Schuyler, 1998; Bodilly, Purnell, Ramsey, & Keith, 1996; Slavin et al., 1996). Having all teachers in a school share a knowledge base, as well as share expectations for teaching and learning, facilitates teaching and learning toward high standards (O'Day & Smith, 1993). Furthermore, researchers and policy makers are increasingly emphasizing the importance of professional development that is embedded in the daily life of teachers (Guskey, 1997; Loucks-Horsley et al., 1998). As a result, they have focused on the school as the logical unit for both planning and implementing ongoing professional development activities (Guskey, 1997; Senge, 1990; Shanker, 1990). Presumably, districts can encourage and support school-level planning of professional development through involvement of teachers.
In the discussion section of this article, in the context of our study findings, we again draw on this literature about the district role in alignment, coordination, planning, and continuous improvement. The focus here is to ground our choice of variables to include in our study in the literature on effective professional development. In the discussion section, we draw on the literature again, this time to situate and explain the findings for our study.
DEFINING FEATURES OF HIGH-QUALITY PROFESSIONAL DEVELOPMENT
Only a few studies link characteristics of professional development with teachers' instruction and student achievement (e.g., Cohen & Hill, 2000; Desimone et al., in press; Fennema et al., 1996; Caret et al., 1999, 2002; Wilson & Ball, 1991; Porter et al., 2000). However, over the past decade, considerable literature has emerged on professional development, teacher learning, and teacher change, which provides guidance about the characteristics for high-quality professional development (e.g., Corcoran, 1995; Darling-Hammond, 1995; Hargreaves & Fullan, 1992; Hiebert, 1999; Kennedy, 1998; Lieberman, 1996; Little, 1993; Loucks-Horsley et al., 1998; Richardson, 1994; Richardson & Placier, in press; Sparks & Loucks-Horsley, 1989; Stiles, Loucks-Horsley, & Hewson, 1996).ivv
From this literature we identified six features of best practice. We identified three features of the structure or design of the activity. The three structural features we focus on are the form or organization of the activity that is, whether the activity is organized as a reform type, such as a study group, teacher network, mentoring, committee or task force, internship, individual research project, or teacher research center, in contrast to a traditional workshop or conference; the duration of the activity, including the total number of contact hours that participants are expected to spend in the activity, as well as the span of time over which the activity takes place; and the degree to which the activity emphasizes the collective participation of groups of teachers from the same school, department, or grade level, as opposed to the participation of individual teachers from many schools.
In addition to these structural features, we identified three dimensions of the substance or core of the professional development experience. The three core features we measured are the degree to which the activity has a content focusthat is, the degree to which the activity is focused on improving and deepening teachers' content knowledge in mathematics and science; the extent to which the activity offers opportunities for active learning, such as opportunities for teachers to become actively engaged in the meaningful analysis of teaching and learningfor example, by reviewing student work or obtaining feedback on their teaching; and the degree to which the activity promotes coherence in teachers' professional development, by incorporating experiences that are consistent with teachers' goals, aligned with state standards and assessments, and encouraging of continuing professional communication among teachers.
Using data from a national probability sample of teachers (reported in Caret et al., 1999, 2002), we found that the three structural featurestype, duration, and collective participationaffect the substance or core of the professional development activities.vi The core featurescontent focus, active learning, and coherencein turn were significantly related to increases in teacher's knowledge and skills and changes in teaching practice.vii We found similar results in a 3-year study of a longitudinal sample of teachers (see Desimone, Porter, Garet, Yoon, & Birman, in press; Porter, Caret, Desimone, Birman, & Suk Yoon, 2000). Thus, there is empirical support that these six characteristics of professional development that we use in this district analysis are significantly related to increasing teachers' knowledge and changing classroom practice.
In addition to the structural and core features of professional development activities, another important factor that affects teachers' professional development opportunities is the extent to which districts target activities toward specific groups of teachers. A common capacity-building strategy districts use is to target resources and attention to schools with low-performing students (Massell, 1998). Teachers of low-achieving students and other special populations of students are more likely than other teachers to have little teaching experience; be working at schools with fewer resources; have larger classes; have behavior, safety, and other nonacademic issues to confront; and have students who are more challenging to teach than more advantaged students (Darling-Hammond, 1997a; U.S. Department of Education, 1999a). Teachers of disadvantaged students often do not feel prepared to meet the needs of their students (U.S. Department of Education, 1999). Thus, these are often the teachers who most need professional development (Corcoran, 1995; Darling-Hammond, 1997b). And although they may be more in need of professional development than others, teachers of students in schools with high rates of poverty and low-achievement or Title I schools sometimes have less opportunity to participate in certain types of professional development than teachers of students in more advantaged schools (U.S. Department of Education, 1998).
District context plays a major role in determining the success of a district's reform efforts. The number of students in a district and the poverty level of students in the district are especially important influences on the capacity of a district to respond to reforms (e.g., Fuhrman et al., 1988; Hannaway & Kimball, 1997; Spillane & Thompson, 1997). We tested the extent to which these factors affected the management, implementation, and quality of professional development activities.
At the time we drew the national probability sample of Eisenhower districts in 1997, approximately 93% of districts received funding from the Eisenhower program, including districts that were part of consortia that participated in the program. Nationally, all large, urban districts are included in the sample (see Appendix).
The data we use in this district study are from computer-assisted telephone interviews (CATI) of the national probability sample of Eisenhower district professional development coordinators, conducted in the spring of 1998. During the interviews, coordinators reported on specific professional development activities that occurred from July 1997 through December 1998; questions about general practices applied to the 1997-1998 school year.
Through a process of stratified random sampling, to allow variation on poverty level, we targeted a total of 400 districts across the country. The probability of a district being chosen for our saniple was proportional to district size (i.e., the number of teachers in the district). Consequently, all of the data are weighted by district size. We obtained survey data from a sample of 363 district Eisenhower coordinators, yielding a response rate of 88%.viii We provide more information about sample design, response rates, sampling probabilities and weights in the Appendix.ix
In this article, we report the results of two analyses* First* we provide descriptive statistics of the prevalence of district strategies nationwide, in Eisenhower-funded districts. Throughout our descriptive analyses of district data, we tested to see where patterns of district support for professional development differed significantly according to district poverty level (defined as the number of children living in poverty in the district) or district size. We divided poverty into three levels (which effectively divide the population into thirds)low (less than 10.9% of children in poverty), medium (from 10.9% to 21.4% of children in poverty), and high (greater than 21.4% of children in poverty). District size is divided into four typessmall (districts with fewer than 250 teachers), medium (districts with between 250 and 1,500 teachers), large (districts with more than 1,500 teachers), and consortia. A consortium is a group of districts, ranging in size from several districts |o several hundred districts, which can sometimes comprise a substantial portion of a state.x In our analyses, we simultaneously control for district size and poverty. We also tested for the interaction between these two variables and report where interaction effects are significant.
Second, we estimate a path model by using a series of ordinary least squares (OLS) regression equations. In our recursive system of simultaneous equations, using separate OLS estimates for each equation is equivalent to full system maximum likelihood (e.g., Greene, 1993) and is a commonly accepted method of path modeling (Bohrnstedt & Knoke, 1982; Li, 1975; Pedhazur, 1982). That is, the variables are ordered from left to right, and variables on the right depend only on variables to their left. We included the full set of control variables in each equation.xi
MEASURES OF MANAGEMENT AND IMPLEMENTATION
To measure district alignment, we created a standardized scale, based on eight items that asked about the extent of alignment of professional development with state and district standards and assessments in mathematics and science. The scale values range from 0 to 5, with 5 indicating that district coordinators report professional development activities to be aligned to a large extent with both state and district standards and assessments, and 0 indicating that district coordinators report not being aligned with standards or assessments at either level.xii
To measure district coordination, we focused on the extent to which districts use funding streams in a strategic way by combining funding sources. We formed a cofunding composite from 0 to 10, indicating the total number of federal (i.e., National Science Foundation and Department of Education) programs with which districts cofunded their Eisenhower-assisted activitiesxiii (of a possible 10).
To measure district continuous improvement efforts, we created a composite consisting of the following variables: (1) whether districts had and used performance indicators, (2) whether districts conducted needs assessments, (3) whether they conducted evaluation, (4) the type of guidance districts provided to schools, and (5) the type of guidance districts provided to professional development providers. We standardized the variables comprising each of these five measures, adjusted the values to be on a 0 to 1 scale, and then summed them to form an overall continuous improvement composite, ranging from 0 to 5. The reliability of the scale is .61 (see Table 1).
Level of Planning
We created a measure of the extent of district-level planning. The scale ranges from 0 to 4, where 4 indicates districts where all planning is done at the district level, and 0 indicates districts where no planning is done at the district level.
Teacher Involvement in Planning
The analysis also includes a measure of the extent of formal teacher involvement in planning through committees. The measure is an additive composite of district reports of teachers' participation through formal committees at the district, school, and cluster levels.xiv The composite is a function of whether teachers are involved in planning at each of the three levels, weighted by the extent to which activities are planned at the three levels (i.e., coordinators indicated whether some, most, or all of their activities were planned at a particular level). The variable ranges from 0 to 3.
MEASURES OF QUALITYxv
The specific measures of the quality of district activities, taken from the district survey, are (1) the percent of the districts' participations in reform professional development activities, from 0 to 100;xvi (2) the average time span of the districts' professional development activities, both reform and traditional, where 1 = less than 1 month, 2 = 1 to 12 months, and 3 = more than 1 year,xvii (3) the extent to which in-district workshops are characterized by collective participation, on a scale from 0 to 1, where 0 indicates no opportunities for collective participation, 0.5 indicates one type of collective participation (e.g., participation by the whole school or by groups of teachers from the school, such, as teachers from the same grade or department), and 1 indicates both types of collective participation (this scale has a reliability of .80, as Table 1 indicates); (4) the number of opportunities for active learning offered in in-district workshops or institutes, on a scale of 0 to 5;xviii and (5) the amount of emphasis the district places on targeting professional development activities to teachers of special populations of students (i.e., teachers from Title I schools, schools with low achievement levels, high-poverty schools,xix special education teachers, and teachers of limited English proficiency students), where 1 = no particular emphasis, 2 = some emphasis, and 3 = strong emphasis. The reliability of this scale is .82 (see Table 1).
Table 1 provides the means, standards deviations, and number of respondents of the variables in the analysis, as well as the internal reliability scores for the scales that we used in the analysis. Table 2 shows the correlations of all of the management, implementation, quality, and contextual variables used in this study. As the correlation coefficients show, there is positive correlation among some of the variables in the model. For example, coordination is significantly and positively correlated with several other management strategies and features of professional development. Most correlations, however, are below .35 and many are below .20. The highest correlation between two features, reform type and time span, supports the logic behind one piece of our hypothesized ordering, specifically that providing professional development in the format of a reform activity allows an extended time span (also, see Caret et al., 1999, 2002). In short, Table 2 shows that there is small-moderate covariation between the dependent and independent variables in our model.
ARE DISTRICTS ENGAGING IN EFFECTIVE MANAGEMENT AND IMPLEMENTATION PRACTICES?
Using our national probability sample of district Eisenhower coordinators, we examine the extent to which districts are engaging in the following strategies, identified in the literature and supported by the Eisenhower program: alignment, coordination, continuous improvement efforts, teacher participation in planning and district-level planning for the provision of high-quality professional development.
District alignment. xx
As Table 3 shows, most districts report having state standards and assessments, district standards and assessments, or both, and that they are aligned. Districts report, however, that alignment is more likely for standards than for assessments. Further, as indicated in Table 8, activities in large districts are significantly more likely to be aligned with state and district standards and assessments than professional development activities in small districts. Poverty-level differences in alignment are not significant.
Table 4 indicates that the Eisenhower program is more likely to cofund with programs funded by the National Science Foundation (NSF) than programs funded by the Department of Education (probably because of NSF's mathematics/science focus). On average districts cofund with 2 out of 10 programs. As Table 8 shows, generally cofunding increases as district size and poverty level increases. The exception is that for consortia, medium-poverty districts cofund less than low-poverty districts.
District continuous improvement efforts.
Fewer than one in five, or about 18%, of teachers are in districts that currently collect data on performance indicators that they have established to guide their professional development efforts (data not shown). Similarly, less than 25% of the nation's teachers are in districts that report being affected by their states' indicators for professional development.
As Table 5 illustrates, the most common methods of needs assessments are teacher surveys (80%) and informal conversations (76%). Nearly every district reports evaluating their professional development activities (93%, data not shown). These evaluations, however, are frequently based on teacher satisfaction surveys (85%) or participation counts (70%). It is less common to use observations of teachers' subsequent classroom practice (71%) or student achievement scores (60%) to evaluate the effectiveness of Eisenhower professional development.
Further, districts provide fewer data-related types of support to schools, such as reports (33%) and evaluations (41%), than other types of support, such as developing plans (81%), developing activities (82%) and making phone calls (79%).xxi
As shown in Table 8, there is significant variation in continuous improvement according to both district size and poverty level. Small districts employ significantly fewer continuous improvement efforts than either consortia or large and medium districts, and medium districts employ significantly fewer continuous improvement efforts than large districts. Similarly, low-poverty districts have fewer continuous improvement efforts than high-poverty districts.
Level of planning.
Table 6 shows that only 7% of teachers are in districts where no Eisenhower-assisted activities are planned at the district level, and 16% are in districts where all activities are planned at the district level.xxii This means that most districts plan at both the district and school levels. There are no significant differences in the extent of district-level planning by district size or poverty.
For all districts, we find that 65% of teachers are in districts in which teachers participate in formal committees at the district level, and 62% of teachers are in districts in which teachers participate in formal committees at the school level (shown in Table 7); this difference is not statistically significant (data not shown). There are no statistically significant differences according to district poverty level and district size in teacher's formal involvement.
District Poverty and District Size
Table 8 shows differences according to district poverty and size on the structural and core features of professional development activities. The level of poverty in a district sometimes is significantly associated with particular patterns of support for professional development activities. Our data from district coordinators indicate that, controlling for other variables, compared with lower poverty districts, higher poverty districts have more teacher participation in reform professional development activities and less participation in traditional activities; offer more types of Eisenhower-assisted activities, both traditional and reform; and place more emphasis on recruiting teachers of special populations of students.
The analysis for district size shows that, controlling for other variables, compared with districts with fewer teachers, the portfolios of professional development activities in districts with larger numbers of teachers have fewer participations in traditional types of professional development (except in low-poverty districts); provide more opportunities for active learning in professional development activities; have activities designed with more opportunities for collective participation; offer activities that span a longer period of time; provide a larger number of different types of activities, both traditional and reform; and place more emphasis on recruiting teachers of special populations of students. In short, large and high-poverty districts are likely to provide higher quality professional development, when controlling for other variables (probably because of differences in capacity and access to funds linked to district poverty status).
HOW DO DISTRICT STRATEGIES RELATE TO THE QUALITY OF PROFESSIONAL DEVELOPMENT ACTIVITIES THAT DISTRICTS PROVIDE?
To analyze how the management and implementation strategies we have identified-those addressing the district's role in alignment, cofunding, continuous improvement, and level of teacher involvement in planning are associated with each other and with features of high-quality professional development, we developed an explanatory model, shown in Figure 1. For coefficients for all variables in the model, see Table 9. The model is an implied logic model, in that we hypothesize a sequence of events. Specifically, as Figure 1 depicts, we assume that districts first build a. vision of professional development through alignment and cofunding, then implement and monitor the vision through planning and continuous improvement efforts. These actions* then result in particular features of the district portfolio of professional development, such as the percentage of teachers in reform types of professional development activities, the average duration of activities, the degree of collective participation, opportunities for active learning, and the district's targeting practices.
We hypothesize this ordering based on both the professional development and systemic reform literature and the logic of the Eisenhower legislation. Specifically, the literature suggests that alignment and coordination activities require districts to be thoughtful about planning and continuous improvement strategies, such as use of data, needs assessment, and evaluation, and that all of these activities can affect the quality of the professional development (e.g., Cohen & Hill, 1998; Corcoran, 1995; Elmore & Burney, 1996; Guskey, 1997; Smith & O'Day, 1991; Webb, 1998).
Similarly, the Eisenhower legislation requires that districts align their activities with state and local content and performance standards and that districts coordinate their Eisenhower activities with other sources of funding for professional development activities.
The literature also suggests that teacher involvement in planning contributes to high-quality professional development (Clark, 1992; Elmore, 1993; Floden et al, 1988; Loucks-Horsley et al., 1998). This is mirrored in the Eisenhower legislation, which requires districts to work with teachers and other school-level staff in planning professional development activities.
Further, research has shown that such continuous improvement mechanisms help focus local priorities and improve the design and implementation of professional development (Fuhrman et al., 1988; Guskey, 1997; Loucks-Horsley et al., 1998). Likewise, the guidelines and legislation for the Eisenhower program also require efforts consistent with the "continuous improvement" paradigm that has permeated all federal programs in recent years, spurred by the Government Performance and Results Act of 1993 (GPRA). The law requires districts to establish goals and objectives in the form of performance indicators, assess the needs of teachers, and use evaluation data to track progress toward these indicators, all of which support a data-based continuous improvement process.
Thus, we ground our model in the logic of the legislation and research, which suggests that having a vision of professional development based on alignment and coordination affects how districts plan and continuously improve their activities, which in turn affect the quality of those activities.
It should be emphasized, however, that components of the system are likely interactive and may occur simultaneously. For example, a reform-oriented district may practice superior vision building and implementation and design professional development activities with more high-quality components and more targeting, all at the same time, because of the district's orientation toward reform. Our data are not longitudinal, so we cannot test the causal ordering. We can, however, identify the strength of relationships among variables. We suggest a logic of events to help to explain how the process of designing and implementing district-provided development might work, but our model should not be considered to exclude the possibility of nonrecursive effects or an alternative temporal ordering.
As described in the methods section, we use ordinary least squares regression (OLS) to analyze the relationships among variables. All paths shown are statistically significant at the .05 level. Because contextual factors may influence the design and implementation of district activities, we have included several district characteristics as control variables in our model: district poverty level, consortium status, the log of the number of teachers,xxiii the interaction of the log of the number of teachers and consortium status,xxiv and cluster status.
As Figure 1 shows, cofunding is the strongest predictor of our indicators of the quality of district-sponsored professional development activities. Cofunding is related directly to increased targeting (b = .17),xxv a higher percentage of teacher participations in reform types of professional development (b = .15), and more collective participation (b = .14); it is indirectly related to more opportunities for active learning and increased targeting through its relationship to increased continuous improvement efforts (b = .16) and more teacher participation in planning (b = .16). Cofunding may provide the level of funding that such high-quality professional development requires. Similarly, alignment of professional development with standards and assessments is significantly related to implementation and structural and core features of professional development. Alignment predicts more participations in reform types of professional development (b = .12), which in turn is associated with a longer span (b = .40). Alignment also predicts more continuous improvement (b = .16). Continuous improvement efforts, in terms of providing guidance, needs assessments, and evaluation, have an association with both increased opportunities for active learning (b = .20) and increased targeting (b = .17). Alignment may serve to clarify goals which in turn makes the system operate in a more coherent and effective manner.
Figure 1 also shows that more district-level planning is related to activities of longer span (b = .11). Teacher participation in planning is related to activities with more opportunities for active learning (b = .20) and more targeting of teachers of special populations of students (b = .16). Perhaps teachers are more likely to recognize and call for professional development with these features shown by the literature to be effective.
In summary, results from the analysis show a relationship between management strategies and what the literature suggests and the Eisenhower, program hypothesizes to be features of high-quality professional development. Our path estimates supported all of the management and implementation strategies described in our model and supported by the systemic reform literature. Further, we conducted a similar analysis of the effect of the management and implementation strategies used by a national probability sample of professional development project directors at institutions for higher education (IHEs). Although the district and IHE models were estimated independently, the results from each analysis are quite consistent, each replicating the other. For details about the IHE model, see Caret et al., 1999. Although the results of the district path analysis support our hypothesized sequential ordering, it should be emphasized that these are cross-sectional data, and other configurations of relationships might also fit the data.
DISCUSSION AND IMPLICATIONS FOR THOSE WHO DESIGN AND FUND PROFESSIONAL DEVELOPMENT
The literature based on case studies of selected districts suggests that district management and implementation strategies can play an important role in the overall quality of the district portfolio of professional development activities. Our study sought to replicate and clarify these suggestive findings on a national probability sample of Eisenhower supported districts. Here we synthesize our findings to draw conclusions about the effectiveness of these strategies.
In interpreting the results, both the strengths and limits of the data should be considered. First, our data are based on self-report surveys. We took a number of steps to maximize the validity and reliability of this national survey data. For example, although the telephone interview data are based on self-reports, most of the data represent an accounting of behaviors, not direct judgments of quality that might be more likely biased in a positive direction (Mullens & Gayler, 1999; Mullens, 1998). In addition, the substantial variation in the responses that districts in our national sample provided to these behavioral items, as well as the consistency in district and teacher responses, provides support for the validity of the data. Further, research has shown that composite indicators have higher validity and reliability than single indicators (Mayer, 1999), and most all of our measures are composites. However, just as case studies cannot make claims about the generalizability of their results, but can explore in depth how things work, surveys cannot provide depth and detail in explaining how relationships operate, but representative survey data do increase generalizability.
Second, our data are based on one respondent per district. There is case-study evidence to suggest substantial within-district variation in accounting of policy directives (e.g., Spillane, 1998). Although clearly our findings would be strengthened by multiple respondents from the same districts, the survey results are, to some extent, cross-validated through case study data that were part of the larger study.xxvi Further, that we found the hypothesized relationships among variables of district strategies and indicators of quality of professional development suggests a reasonableness to the data.
Third, our dependent variables are features of professional development. Although it would be useful to have student achievement gains as the dependent variable, that was not feasible for both cost and logistical reasons in a national probability sample study. Instead, we chose our dependent variables to represent the factors that the literature showed were related to student achievement (e.g., Cohen & Hill, 1998). All of these strengths and weaknesses should be kept in mind when interpreting the results.
BUILDING A VISION OF PROFESSIONAL DEVELOPMENT: ALIGNMENT AND COORDINATION
Alignment was significantly related to several structural and core features of professional development Our model showed that districts that align professional development with standards and assessments are more likely to offer reform types of activities and are more likely to engage in continuous improvement efforts, which are in turn related to increased opportunities for active learning.
Although there are several ways that districts might approach alignment, including building consensus around goals and philosophy of instruction (Massell, 1998), we focused our measure on the alignment of professional development with standards and assessments. Our descriptive results indicate that most district Eisenhower coordinators report aligning professional development with state and district standards and assessments.xxvii There is, however, more alignment with standards than with assessments. This may be because more states have standards than assessments, and even if assessments exist, they may not yet be well aligned to the standardsthe ESEA requirement for aligned assessments had not yet gone into effect at the time of our data collection. Also, assessments have proven more difficult to develop than standards, and often states and districts first establish standards and then seek to revise their assessments to be aligned with the newly developed standards.
District coordinators report that professional development is as aligned with state standards and assessments as with the more immediate district standards and assessments. One possible explanation for this is that district policy tools tend to reflect state policy tools, so they are seen as interchangeable. It may also reflect greater stakes associated with state standards and assessments.
Finally, the reported levels of alignment in our study are based on the respondent's conception of alignment. Alignment is a complex variable which can range from very low levels to near perfect alignment. Respondents may have been generous in their reported levels of alignment. Apparently, however, the variance in the reported levels of alignment is valid because the variable worked well in our model.
Coordination, defined in our study as cofunding, is one of the most important variables in our model for explaining variance in the quality of professional development. We found that coordination plays an important role in the implementation, management, structural, and core features of professional development. Districts that engage in more cofunding with other programs tend to support a greater proportion of reform types of activities than districts that engage in less cofunding, provide more opportunities for collective participation, and involve teachers more widely in planning, which in turn is related to increased targeting of teachers of special populations of students.
Further, more coordination is associated with the use of more continuous improvement strategies, which, in turn, are related to an increased number of opportunities for active learning, and increased targeting. Our descriptive data help interpret the results. For districts, the higher levels of cofunding that occurred with National Science Foundation programs than other federal programs highlights the importance of having a subject-area focus in common to promote cofunding.
Implications: Alignment and coordination.
These results support the notion that building a vision of professional development through alignment, and having a critical mass of funds available, made possible through cofunding, are instrumental factors in fostering the provision of high-quality professional development activities. Our findings suggest that the quality of professional development activities provided by districts could be improved by increasing alignment of district activities with standards and assessments and encouraging districts to coordinate through cofunding professional development with other programs. Although we find little evidence of widespread cofunding, and moderately high levels of reported alignment with standards, case-study research suggests the need to characterize alignment and coordination on multiple dimensions to understand why certain practices are effective. For example, Spillane and Thompsons (1997) study of nine school districts found that most district did not have the deep, substantive alignment intended by systemic reform but instead had superficial alignment strategies.
In our study, professional development aligned with standards and assessments serves as an indicator for the larger construct of alignment and cofunding as a specific indicator for the larger construct of coordination. These measures are useful in identifying national patterns of such practice, and they complement the in-depth case-study analysis necessary to understand the full array of complex district coordination and alignment strategies.
Further, case-study research suggests that some districts may recognize the need to provide more effective professional development and could benefit from using mechanisms such as alignment and coordination to achieve this goal. For example, studies of several districts have shown that although traditional formats of professional development are easier to manage, implement, and monitor (Little, 1993), districts are aware of the limitations of these formats and are increasing their interest in pursuing less traditional formats for professional development, such as networks and mentoring (Massell, 1998). These districts might find supporting higher quality professional development more within their reach if they invested more heavily in the particular types of alignment and coordination that have been shown here to be associated with the provisions of high-quality professional development. In short, these findings support the efficacy of having those who fund and design professional development emphasize the cofunding professional development activities with multiple sources of money.
In our model, continuous improvement efforts have a significant relationship to the structural and core features of professional development activities provided by districts. Our data show that continuous improvement efforts are significantly related to increased opportunities for active learning and increased targeting of teachers of special populations of students (e.g., at-risk students). Perhaps districts focus on including active learning techniques and targeting special teachers in response to the needs assessments and evaluations provided by teachers, which are part of continuous improvement strategies.
Our descriptive data show that districts report moderately high levels of engagement in most continuous improvement efforts, such as conducting needs assessments and evaluations; few districts however, use district performance indicators in designing their professional development activities. In her study of 22 districts, Massell (1998) identified a growing emphasis on the use of performance and other data to plan professional development and other reforms and on districts assisting schools to use their own data to assess teacher needs. But our data show that this is not a national trend. Most districts do not have performance indicators, remain unaware of the federal requirement to develop them, and are unaware of their purpose in guiding the continuous improvement process. Further, although many districts offer particular types of guidance and support to schools and professional development providers, districts offer little guidance related to the use of data.
Similarly, although almost all districts conduct some form of needs assessments and evaluations, these processes are typically based on teacher satisfaction surveys, informal conversations with teachers, or participation counts. This is consistent with earlier research conducted by Little (1989). In a study of 30 districts in one state, she found that evaluation was thin and that needs assessments were oriented toward individual teachers instead of addressing the needs of entire faculties or groups of teachers or departments.
Teachers may evaluate the quality of professional development against their own individual goals but not against the goals of the district. Further, teachers may not be able to identify their own weaknesses, and even if they could they may not be able to identify appropriate measures to address the weaknesses. District effort at continuous improvement would be stronger if they were to focus less on teacher perceptions and reactions and place greater emphasis on the outcomes of professional development for teachers and students (Guskey, 1997).
More objective ways of assessing needs and evaluating the outcomes of professional development, such as observations of teachers or measures of student achievement, may be less commonly used for needs assessment and evaluation in part because of the difficulty of obtaining these measures (e.g., student achievement) and linking them to professional development activities. Also, most districts do not have the resources to use these approaches effectively. Studies that link professional development with student outcomes are complex and would require costly long-term data collections and an evaluation capacity that many districts do not have. There are simply too many intervening variables between professional development experiences and subsequent gains in student achievement to make such studies feasible at the local level.
Implications: Continuous improvement.
Although continuous improvement efforts proved to be significantly related to the quality of professional development that districts provide, district efforts in this area are not as strong as they could be. If using data to make decisions is to be a serious endeavor, then districts may need assistance in building capacity to foster continuous improvement effortsfor example, in learning how to develop well-constructed and useful indicators, how to collect meaningful data, and how to interpret and use the indicators as part of a system of continuous improvement.
Also, rather than focusing solely on teacher feedback, it seems appropriate and potentially effective for local evaluation of professional development to focus on the degree to which the professional development is characterized by well-implemented attributes of high-quality professional developmentfor example, a focus on content, active learning, and coherence, delivered with sufficient duration and collective participation. Such evaluations might be based on a variety of sources of data, including well-designed surveys of participating teachers and the observation of teachers to assess the extent to which they have made appropriate changes in their instruction. Although the assessment of student achievement might be used as part of a balanced evaluation of professional development, local evaluation efforts should not focus solely on assessing the effects of professional development on student achievement. Although clearly there needs to be more research that looks at the relationships between features of professional development and gains in student achievement, these types of evaluations are better done as a part of well-designed major research studies.
Thus, designers and funders of professional development should encourage the appropriate use of evaluation, assessment, and other types of continuous improvement. However, districts need to build capacity in this area to enable them to collect and interpret useful data to help them improve their professional development programs.
Planning: District, School, and Teacher Involvement
Our model shows that activities planned at the district level span a longer period of time than activities not planned at the district level and that teacher participation in planning is associated with more opportunities for active learning and more targeting of teachers of at-risk students.
Teacher participation, at any level of planning, is associated with positive aspects of professional development. District-level planning is related to a significantly longer span, but this may be because the district is in a better position to sponsor yearlong activities; analysis of our data and information from our case studies shows that high-quality professional development can be planned at either the district or school level.xxviii
Teacher involvement in planning professional development at the district level can help to ensure that topics and learning activities in professional development programs address areas of knowledge and skills that are relevant to teachers district-wide. This can be particularly important for teachers who may be the only ones teaching a particular subject in their school (e.g., physics teachers in small high schools). At the same time, teacher participation in planning at the school level and professional development that takes place at the school allow for more coherent professional development that is closely tied to the needs of teachers in a particular, school and thus potentially more relevant to classroom practice. Also, teacher participation in school-level planning might increase the likelihood that a critical mass of teachers will participate (e.g., McLaughlin & Talbert, 1993). In our district model, however, we did not find a relationship between level of planning or teacher participation in planning and the collective participation of all teachers or groups of teachers at the school.xxix In other words, planning for professional development at the school level, or occurrence of the professional development activity at the school site, does not necessarily mean that the professional development involves the collective participation of all teachers or groups of teachers at the school.
Implications: Teacher involvement in and level of planning.
Although district-level planning and teacher participation in planning are related to the provision of high-quality professional development, how this process works is unclear. Teacher participation occurs at all levels of planning, and just because an activity is planned at the district or school level does not mean it is implemented at that level. Research on professional development suggests that meeting the needs of individual teachers and meeting school-wide needs are both important goals of professional development. For districts, having active participation of teachers can be important in developing teacher support for the professional development program, as well as for ensuring that activities focus on methods and approaches that the teachers think are important. We need a better understanding, however, of how teacher participation in planning addresses individual and school needs and how it affects the core and structural characteristics of activities to better assess the importance of school level implementation and planning of professional development.
Contextual Factors: District Size and Poverty
Our descriptive data showed that, generally, large districts, and to some extent high poverty districts, are more likely to manage their professional development activities better and to provide higher quality professional development than are other districts. Specifically, large districts are more likely than small districts to align their professional development with standards and assessments, to cofund their projects, and to have a greater commitment to continuous improvement. Large districts also provide activities of longer duration, with more opportunities for collective participation and active learning. High-poverty districts cofund more with National Science Foundation and Department of Education programs and employ more continuous improvement efforts than low-poverty districts.
Large districts may be more likely than other districts to align their professional development with standards and assessments because large districts have a more developed and sophisticated method for integrating state and district reforms with professional development activities. Previous research has shown that administrators often believe that they need more information on how to link professional development with standards (Celebuski & Farris, 1998), and this lack of information may be more prevalent in small than in large districts. It also may be that districts with more teachers have a greater need to create an organized strategy for the design of their professional development activities and thus are more-likely than smaller districts to use state and district standards and assessments for this purpose.
Both large districts and consortia cofund with other programs significantly more than smaller districts do, perhaps because larger districts may be able to incorporate effective collaborative practices due to economies of scale (e.g., investment in collaborative efforts is less expensive as the number of participants and programs increases) and also because they have larger, better infrastructures for planning and delivering professional development. The resources available to districts clearly differ dramatically and have a profound effect on a district's, ability to achieve its goals (Elmore, 1993). Large districts may have funds from multiple sources that increase opportunities for coordination and cofunding. Also, large districts are more likely to have subject-area specialists that are integrated into the mathematics/ science community. By contrast, smaller districts may have fewer programs and personnel with which to collaborate. Smaller districts also may not have the capacity or incentive to work across programs in the way that often becomes necessary in larger districts.
For example, Firestone and Fairman (1998), in their study of five school districts in two states, found that district configuration, including size, influenced differences in approaches to district policies and professional development. For example, the larger districts had more capacity to work with teachers and bring in outside experts than smaller districts did. Similarly, Florian, Hange, and Copeland's (2000) study of 16 districts in 13 states found that communication with outside constituencies to build shared and aligned goals helped increase their capacity to support local reform.
The relationship between cofunding and district size also might be related to larger districts having more access to expertise. Several district case studies have found that the district's choice of where to focus funds is related to the district administrator's knowledge of reforms (Price, Ball, & Luks, 1995; Spillane, 1998; Spillane & Thompson, 1997). Further, Hannaway and Kimball's (1997) study of a nationally representative sample of administrators in 50 states showed that smaller districts had less understanding of standards-based reform and reported less progress on standards-based reform goals. Hannaway and Kimball (1997) concluded that district knowledge of reform influenced the ways that staff interpreted and implemented policies, the type and level of support they gave to the policies, and the way in which they allocated resources.
Districts with a higher proportion of students in poverty are also somewhat more likely to cofund with other federal programs. This might be because high-poverty districts tend to receive money from multiple federal programs whose formulas are often designed to favor districts with larger numbers of children from low-income families. The existence of multiple sources of funding with similar programmatic goals may create greater opportunities for cofunding and create opportunities for more collaborative efforts in employing strategies for making multiple funding streams support each other and integrate professional development across programs to aid in the education of students with special needs.
Large and high-poverty districts and consortia engage in more continuous improvement than other districts perhaps partly because large districts have more staff and a more comprehensive professional development program, which requires more systematic monitoring and evaluation. Similarly, high-poverty districts are more likely to have support from other federal programs, such as Tide I, Part A, which also emphasize the use of indicators, needs assessments, and evaluation.
Both large and small districts have significantly more school-level planning than consortia, which makes sense given that consortia would be more likely than single districts to plan at the district level because consortia by definition are a group of districts working together. Further, because consortia may be more focused on across-district planning, school-level planning may not be emphasized as much as in single districts.
Implications for district size and poverty.
Our analyses indicate that larger districts provide higher quality professional development and, to some extent, so do higher poverty districts. Consortia that tie together several small districts into one unit for providing professional development also seem, on average, more effective than small districts.xxx The greater effectiveness in providing professional development may be explained, in part, by their better infrastructure, larger district staff, and, in part, by their greater amounts and sources of funds. Large and high-poverty districts also may have a greater variety of funding sources, increasing opportunities for cofunding and perhaps creating a complexity that demands a commitment to continuous improvement. If this is the case, perhaps small districts need more guidance to help them organize and combine their funding streams (although they are likely to have fewer funding streams than large districts) and collaborate with other professional development providers in their district.
Previous studies of district capacity inform our findings. Several studies of school districts describe capacity as having many dimensions, which include not only financial and personnel resources but also district culture and values, relationships, networks, and trust. For example, Spillane and Thompson (1997) found that much of the variation in district implementation of the state's vision of reform was due to differences in district capacity. Although they found that time and financial resources were linked to a district's ability to implement reform, they found other factors related to capacity that also influenced district success. They identify three dimensions of capacityhuman capital, social capital, and financial resourcesand discuss in depth how each plays out at the district level. For example, they found that linkages to knowledge sources outside the district enabled some districts to build their knowledge about state reforms and that district capacity was linked to the district's ability to learn about new reforms and help teachers and others in the district learn the ideas behind the reform. Similarly, Firestone and Fairman (1998) define capacity in terms of the ability to mobilize personnel, address problems, monitor reform, and include educators in decision-making processes, and Chrispeels (1997) describes capacity in terms of district commitment to reform.
The state context also plays a role in determining district capacity. For example, Goertz, Massell, and Chun (1998) noted patterns in district activities according to whether the state had strong or weak accountability systems (defined by prescriptiveness, rigor, alignment of the system with other reforms and the stability of the system). In addition, in their study of 40 districts in California, Marsh and McCabe (1998) found that districts already had in place most of the policy components of a new state policy initiative designed to drive standards-based reform, raising issues about the relationships between state and district policies. Similarly, Spillane (1996) argues that stronger state policy provides districts with more opportunities for district policy making, which results in an increase in instructional guidance to teachers. Our study was not designed to address the state role in shaping district strategies but to clearly state that context can play an important role in influencing district actions.
Given the empirical support we found in our national sample of Eisenhower-supported districts for the importance of district capacity, and the complexity of understanding its effects in different contexts, as indicated by the case-study research, the issue of district capacity is one that should be closely monitored in future research. More information is needed on the characteristics and conditions that give some districts the capacity to provide high-quality professional development and the mechanisms available to build and strengthen such capacity. Results reported here are a start.
The provision of high-quality professional development by districts may not completely solve the problem of generally low-quality professional development because participation in professional development remains primarily the decision of individual teachers, and there are many providers of professional development other than districts. Nevertheless, districts could go a long way in developing high-quality professional development activities, and effective district management and implementation strategies could be applied by other providers (e.g. university providers).
Our analyses of a national probability sample31 of Eisenhower district professional development coordinators shows that certain management and implementation strategies are related to the quality of professional development activities that districts provide to teachers. Specifically, alignment with standards and assessments, district cofunding, continuous improvement efforts such as establishing indicators and conducting needs assessments and evaluation, and teacher involvement in district-level planning predict the core and structural features of professional development activities. Further, large districts, and to some extent high-poverty districts, engage in these strategies more than others. These findings suggest that policy supports to increase the capacity of all districts to use these management and implementation strategies might help to make professional development more effective for teachers and, ultimately, for their students.
APPENDIX: SAMPLE DESIGN, RESPONSE RATES, SAMPLING PROBABILITIES, AND WEIGHTS FOR DISTRICT COORDINATOR INTERVIEWS
SAMPLE DESIGN AND RESPONSE RATES FOR DISTRICT COORDINATOR INTERVIEWS
Designing the sample.
We based the district sampling frame for the 1997-98 Teacher Activity Survey on the Common Core of Data (CCD), maintained by the National Center for Education Statistics. At the time we selected the sample, the most recent year for which complete CCD data were available was 1992-93. According to the CCD, there were 14,645 districts in the 1992-1993 school year. The percentage of students in poverty is based on 1990 Census data, aggregated to the district level by the National Center for Education Statistics. The cut points for the three poverty status groups were chosen to divide the national population of teachers into thirds.32
The CCD data indicated that district size (as measured by the number of teachers) is highly skewed. About half of the districts in the country have fewer than 60 teachers, but these districts as a group account for less than 10% of the teachers. On the other hand, just over 5% of the districts in the country have 500 or more teachers, but such districts as a group account for more than half of the teachers in the country.
Because district size is highly skewed, a simple random sample would contain many small districts representing very few teachers. Thus, we drew the sample of districts with probability proportional to district size, separately within each of the three poverty strata, using the number of teachers employed as the measure of size. This approach allowed us to obtain efficient estimates of district characteristics, weighting districts in proportion to the number of teachers employed.
Within each stratum, we selected with certainty all districts with 5,000 or more teachers4 low-poverty districts, 10 medium-poverty districts, and 12 high-poverty districts.
To obtain sufficiently precise estimates of district characteristics, we planned to conduct interviews with 400 district coordinators. Based on pilot interviews in a small sample of districts, we learned that, in some large districts , Eisenhower funds are divided among subdistricts. When districts divide funds among subdistricts, we concluded that it might be necessary to conduct separate interviews and obtain separate activity lists within each sub-district. In such cases, each subdistrict would operate, in effect, as a separate district. To estimate the number of subdistrict interviews that might be required, we assumed that subdistrict interviews would occur in only very large districts (i.e., the 26 districts with more than 5,000 teachers), and we assumed that one subdistrict interview would be required per 5,000 teachers. These calculations led us to estimate that the 26 certainty districts in our sample might generate 53 interviews altogether27 more than would be required without subdistricts.
Because we desired an overall sample size of 400 interviews, we set a total sample size of 373 districts, to accommodate the anticipated 27 additional subdistrict interviews. Because we planned to select 26 districts with certainty, this left 347 districts to be drawn with probability proportional to size. We allocated these 347 districts to the three strata in proportion to each stratum's total number of teachers in districts with fewer than 5,000 teachers. This procedure yielded a sample size of 140 low poverty districts, 129 medium-poverty districts, and 104 high-poverty districts.
Screening districts and scheduling interviews.
After drawing an initial sample of districts, we contacted the states in which the sample districts were located to check whether the sampled districts participated in the Eisenhower program and to obtain the name and telephone number of the district Eisenhower coordinator. About 7% of districts did not participate. We replaced nonparticipating districts that fell into the sample with randomly drawn districts of similar size and poverty status.
Finally, we contacted the districts drawn into the sample to invite them to participate in the evaluation. If a district refused, we replaced the district with a randomly drawn district of similar size and poverty status.
In all, we contacted 409 districts that received Eisenhower funds and were thus eligible to participate in the study.
When we contacted each sampled district, we asked the district if it participated as an individual district or as a member of a consortium.
For districts that were members of consortia, we treated the consortium rather than the district as the unit of analysis. For each consortium included in our sample, we obtained a list of the districts participating in the consortium, along with the number of teachers each district employed. We then used these data to determine the probability with which each consortium was drawn.
Most of the districts with enrollments below 1,500 that were drawn into the sample were members of consortia.
Response rates for district interviews.
We attempted to schedule and complete an hour-long computer-assisted telephone interview (CATI) with each of the 439 administrators identified through the screening process. Of these 439 potential interviews, we completed 386, for an overall interview response rate of 88%.
Of the completed interviews, 32 were in districts in which the program was administered by separate organizational units.
In our analyses, we collapsed the multiple interviews conducted in each of these districts into a single district-level response for each item.
Altogether, we completed interviews in 369 districts and consortia, of the 419 districts and subdistricts in which we attempted interviews. This produces a district-level response rate of 88%. This response rate, based on the number of responding districts (369 of 419), is nearly identical to the response rate based on the number of interviews completed (386 of 439), which we discussed above. Of the 369 completed interviews, 6 were unreadable due to a malfunction of the computer-assisted Telephone interviews (CATI) software, leaving 363 in our basic analytic sample.
We examined the response rates for the district telephone interviews to determine whether response rates differ by district size or poverty. We observe a moderate-size effect: Large districts and consortia were somewhat more likely to complete a coordinator interview than small districts (p < .05; see Table A.1). We did not observe an effect of district poverty on response rates.
SAMPLING PROBABILITIES AND WEIGHTS FOR THE DISTRICT COORDINATOR INTERVIEW
The final weights used in our analyses of the district coordinator interview data reflect two components: a sampling weight, designed to incorporate the fact that districts in our sample were selected with unequal probabilities, and a weight equal to the number of teachers in each district, included so that our reported results represent the role of the district in proportion to the number of teachers in the districts in which the program operates.
As discussed previously, 26 districts (those with 5,000 or more teachers) were included in the national profile with certainty; the remaining sample of districts was drawn with probability proportional to size. The final weights differ for the noncertainty and certainty districts, and thus we will consider each in turn.
In a sample drawn with probability proportional to size, the probability of selection for district k can be expressed:
where n is the number of districts in the national sample, Tk is the number of teachers in district k, and T is the total number of teachers in all districts in the population.
The sampling weight for district k (Wk) is given by the inverse of the sampling probability:
The final district weight involves the product of the sampling weight wk and the number of teacher Tk
Thus, for noncertainty districts, the combination of sampling with probability proportional to size and weighting by the number of teachers produces equal weights for all districts.
For each district drawn with certainty, the sampling weight (i.e., the inverse of the selection probability) is one. Thus, the final weight for each certainty district is simply the number of teachers in the district, Tk.
METHODS FOR DISTRICT HYPOTHESIS TESTS AND STANDARD ERROR ESTIMATION
In the analyses of the district coordinator interview data reported in this paper, the significance tests are computed according to the assumptions of conventional simple random sampling. These tests do not take into account the fact that there is modest variability in the weights (i.e., the largest certainty districts have larger weights than the other districts in the sample).Taking the variability in the weights into account would produce standard errors slightly larger than those upon which the reported significance tests are based.
We have rerun the analyses incorporating the variability in weights, and the significance test results are nearly identical to those reported here.
This research was conducted under Contract Number EA97001001, U.S. Department of Education, Planning and Evaluation Service. We gratefully acknowledge Elizabeth Eisner (our project officer), Alan Ginsburg (director of the Planning and Evaluation Service), and . Ricky Takai (division director) for their contributions to the conceptualization and design of the study. The views expressed herein are those of the authors. No official endorsement by the U.S. Department of Education is intended or should be inferred.
Birman, B., Desimone, L., Caret, M., & Porter, A. (2000). Designing professional development that works. Educational Leadership, 57(8), 28-33.
Bodilly, S., Keltner, B., Purnell, S., Reichardt, R., & Schuyler, G. (1998). Lessons from the New American Schools' scale-up phase: Prospects for bringing designs to multiple schools. Santa Monica, CA: RAND.
Bodilly, S., Purnell, S., Ramsey, K., & Keith, S. J. (1996). Lessons from New American Schools Demonstration Corporation's demonstration phase. Santa Monica, CA: RAND.
Bohrnstedt, G., & Knoke, D. (1982). Statistics for social data analysis (2nd ed.). Itasca, IL: F.E. Peacock Publishers.
Bollen, K. (1989). Structural equations with latent variables. New York: Wiley.
Celebuski, C., & Farris, E. (1998). Status of education reform in public elementary and secondary schools: Principals' perspectives [Statistical Analysis Report]. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.
Chrispeels, J. H. (1997). Educational policy implementation in a shifting political climate: The California experience. American Educational Research Journal, 34(3), 453481.
Clark, C. M. (1992). Teachers as designers in self-directed professional development. In A. Hargreaves (Ed.), Understanding teacher development (pp. 75-84). New York: Teachers College Press.
Cohen, D. K. 1995). What is the system in systemic reform? Educational Researcher, 24(9), 11-17.
Cohen, D. K., & Hill, H. C. (2000). Instructional policy and classroom performance: The mathematics reform in California. Teachers College Record, 102(2), 294-343.
Cohen, D. K., & Hill, H. C. (2001). Learning policy: When state education reform works. New Haven, CT: Yale University Press.
Cohen, D. K., & Spillane, J. (Eds.). (1992). Policy and practice: The relations between governance and instruction. Washington, DC: American Educational Research Association.
Corcoran, T. B. (1995). Transforming professional development for teachers: A guide for state policymakers. Washington, DC: National Governors' Association.
Desimone, L., Porter, A. C., Caret, M. S., Yoon, K. S., Birman, B. F. (in press). Does professional development change teaching practice? Results from a three-year study. Paper presented at the American Education Research Association, New Orleans, LA.
Darling-Hammond, L. (1995). Changing conceptions of teaching and teacher development. Teacher Education Quarterly, 22(4), 9-26.
Darling-Hammond, L. (1997a). Doing what matters most: Investing in quality teaching. New York: National Commission on Teaching & America's Future.
Darling-Hammond, L. (1997b). The right to learn: A blueprint for creating schools that work. San Francisco: Jossey-Bass.
Elmore, R. F. (1993). The role of local school districts in instructional improvement. In S. H. Fuhrman (Ed.), Designing coherent educational policy: Improving the system (pp. 96-124), San Francisco: Jossey-Bass.
Elmore, R. F. (1997). Investing in teacher learning: Staff development and instructional improvement in Community School District #2, New York City. Paper prepared for the National Commission on Teaching and America's Future, supported by the Consortium for Policy Research in Education.
Elmore, R. F., & Burney, D. (1996, March). Staff development and instructional improvement: Community District 2, New York City. Philadelphia: Consortium for Policy Research in Education.
Elmore, R. F., & Burney, D. (1999). Investing in teacher learning: Staff development and instructional improvement. In L. Darling-Hammond & G. Sykes (Eds.), Teaching as the learning profession: Handbook of policy and practice. San Francisco: Jossey Bass.
Fennema, E., Carpenter, T. P., Franke, M. L., Levi, L., Jacobs, V. R., & Empson, S. B. (1996). A longitudinal study of learning to use children's thinking in mathematics instruction. Journal for Research in Mathematics Education, 27(4), 403-434.
Firestone, W. A., & Fairman, J. C. (1998). The district role in state assessment policy: An exploratory study. Rutgers Graduate School of Education. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
Floden, R. E., Porter, A. C., Alford, L. E., Freeman, D. J., Irwin, S., Schmidt, W. H., & Schwille, J. R. (1988). Instructional leadership at the district level: A closer look at autonomy and control. Educational Administration Quarterly, 24(2), 96-124.
Florian, J., Hange, J., & Copeland, G. (2000). The phantom mandate: District capacity for reform, McREL at Aurora, CO; AEL at Charleston, WV; SEDL at Austin, TX. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Fuhrman, S., Clune, W., & Elmore, R. (1988). Research on educational reform: Lessons on the implementation of policy, Teachers College Record, 90(2), 237257.
Fullan, M. G. (1993). Why teachers must become change agents. Educational Leadership, 50(6), 12-17.
Fullan, M. G. (1996). Turning systemic thinking on its head. Phi Delta Kappan, 77(6), 420-423.
Garet, M., Porter, A., Desimone, L., Birman, B., & Yoon, K. S. (2002). What makes professional development effective? Analysis of a national sample of teachers. American Education Research Journal, 38 (3), 915-945.
Garet, M., Birman, B., Porter, A., Desimone, L., & Herman, R. (1999). Designing effective professional development: Lessons from the Eisenhower program. Washington, DC: U.S. Department of Education.
Goertz, M., Massell, D., & Chun, T. (1998). District response to state accountability systems. CPRE, University of Pennsylvania and University of Michigan; The RAND Corporation. Paper presented at the annual meeting of the Association for Public Policy Analysis and Management.
Grant, S. G., Peterson, P. L., & Shojgreen-Downer, A. (1996). Learning to teach mathematics in the context of systemic reform. American Educational Research Journal, 33(2), 502-541.
Greene, W. (1993). Econometric analysis. New York: Macmillan.
Guskey, T. R. (1997). Research needs to link professional development and student learning. Retrieved from http://www.nsdc.org/library/jsd/jsdgusk.html
Hannaway, J., & Kimbali, K. (1997). Reports on reform from the field: District and state survey results. Washington, DC: U.S. Department of Education.
Hanushek, E., & Jackson, J. (1977). Statistical methods for social scientists. New York: Academic Press.
Hargreaves, A., & Fullan, M. G. (1992). Understanding teacher development. London: Cassell.
Hiebert, J. (1999). Relationships between research and the NCTM standards. Journal for Research in Mathematics Education, 30(1), 3-19.
Kennedy, M. M. (1998). Form and substance in in-service teacher education (Research monograph no. 13). Arlington, VA: National Science Foundation.
Knapp, M. S., Zucker, A. A., Adelman, N. E., & St. John, M. (1991, February). The Eisenhower Mathematics and Science Education Program: An enabling resource for reform (Summary Report). Menlo Park, CA: SRI International.
Li, C. C. (1975). Path analysis: A Primer. Pacific Grove, CA: Boxwood Press.
Lieberman, A. (Ed.). (1996). Practices that support teacher development: Transforming conceptions of professional learning. In M. W. McLaughlin & I. Oberman (Eds.), Teacher learning: New policies, new practices (pp.185-201). New York: Teachers College Press.
Little, J. W. (1982). Norms of collegiality arid experimentation: Workplace conditions of school success. American Educational Research Journal, 19(3), 325340.
Little, J. W. (1989). District policy choices and teachers' professional development opportunities. Educational Evaluation and Policy Analysis, 11(2), 165179.
Little, J. W. (1993). Teachers' professional development in a climate of educational reform. Educational Evaluation and Policy Analysis, 75(2), 129-151.
Little, J. W., Gerritz, W. H., Stern, D. S., Guthrie, J. W., Krst, M. W., & Marsh, D. D. (1987). Staff development in California: Public and personal investment, program patterns, and policy choices. San Francisco: Far West Laboratory for Educational Research and Development.
Loucks-Horsley, S., Hewson, P. W., Love, N., & Stiles, K. E. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin.
March, D. D., & McCabe, L. L. (1998). School restructuring: From state policy to district superintendents. Paper presented at the annual meeting of the American Education Research Association, San Diego, CA.
Marsh, J. A. (2000). Connecting districts to the policy dialogue: A review of literature on the relationships of districts with states, school, and communities (A CTP Working Paper). Seattle, WA: Center for the Study of Teaching and Policy.
Massell, D. (1998). State strategies for building local capacity: Addressing the needs of standards-based reform. Philadelphia: Consortium for Policy Research in Education.
Mayer, D. P. (1999). Measuring instructional practice: Can policymakers trust survey data? Educational Evaluation and Policy Analysis, 27(1), 2945.
McLaughlin, M. W., & Talbert, J. E. (1993). Contexts that matter for teaching and learning. Stanford, CA: Center for Research on the Context of Secondary School Teaching.
Mullens, J. (1998). Validating teacher responses on instructional practice self-report surveys: Findings from a recent field-test. Paper prepared for the Association for Public Policy Analysis and Management Research Conference, New York.
Mullens, J. E., & Gayler, K. (1999). Measuring classroom instructional processes: Using surveys and case study field test results to improve item construction. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.
Murphy, J., & Hallinger, P. (1988). Characteristics of instructionally effective school districts. Journal of Educational Research, 81(3), 175-181.
Newmann, F. M., & Associates. (1996). Authentic achievement: Restructuring schools for intellectual quality. San Francisco: Jossey-Bass.
Newmann, F. M., King, M. B., & Rigdon, M. (1997). Accountability and school performance: Implications from restructuring schools. Harvard Educational Review, 67(1), 41-4.
O'Day, J., Goertz, M. E., & Floden, R. E. (1995). Building capacity for education reform (CPRE Policy Brief). New Brunswick, NJ: Consortium for Policy Research in Education.
O'Day, J. A., & Smith, M. S. (1993). Systemic reform and educational opportunity. In S. H. Fuhrman (Ed.), Designing coherent educational policy: Improving the system. San Francisco: Jossey-Bass.
Pedhazur, E. (1982). Multiple regression in behavioral research. Fort Worth, XX; Holt, Rinehart & Winston.
Porter, A. C., Caret, M. S., Desimone, L., Birman, B. F., Yoon, K. S. (2000). Does professional development change teaching practice? Results from a three-year study. Washington, DC: U.S. Department of Education.
Price, J., Ball, D., & Luks, S. (1995). Marshaling resources for reform: District administrators and the case of mathematics. East Lansing, MI: National Center for Research on Teaching Learning.
Richardson, V. (Ed.). (1994). Teacher change and the staff development process: A case in reading instruction. New York: Teachers College Press.
Richardson, V., & Placier, P. (in press). Teacher change. In V. Richardson (Ed.), Handbook of research on teaching (4th ed.). New York: Macmillan.
Rosenholtz, S.J. (1991). Teacher's workplace: The social organization of schools. New York: Teachers College Press.
Schlechty, P., & Whitford, B. L. (1983). The organizational context of school systems and the functions of staff development. In G. Griffin (fed.), Staff development: 82nd yearbook of the National Society for the Study of Education (pp. 52-91). Chicago: University of Chicago Press.
Senge, P. M. (1990). The fifth discipline: The art and practice of learning organization. New York:
Shanker, A. (1990). Staff development and the restructured school. In B.Joyce (Ed.), Changing school culture through staff development. Alexandria, VA: Association for Supervision and Curriculum Development.
Slavin, R. E., Madden, N. A., Dolan, L. J., Wasik, B. A., Ross, S., Smith, L., & Dianda, M. (1996). Success for all: A summary of research. Journal of Education for Students Placed at Risk, 1(1), 41-76.
Smith, M. S., & O'Day, J. (1991). Systemic school reform. In S. H. Fuhrman & B. Malen (Eds.) The politics of curriculum and testing: The 1990 yearbook of the Politics of Education Association (pp. 233-267). Bristol, PA: Falmer Press.
Sparks, D., & Hirsh, S. (May 24, 2000). Strengthening professional development: A national strategy. Education Week; 42-45.
Sparks, D., & Loucks-Horsley, S. (1989). Five models of staff development for teachers. Journal of Staff Development, 10(4), 40-57.
Spillane, J. P. (1996). School district matter: Local educational authorities and state instructional policy. Educational Policy, 10(1), 63-87.
Spillane, J. P. (1998). State policy and the non-monolithic nature of the local school district: Organizational and professional considerations, American Educational Research Journal, 35(1), 33-63.
Spillane, J. P., & Jennings, N. E. (1997). Aligned instructional policy and ambitious pedagogy: Exploring instructional reform from the classroom perspective. Teachers College Record, 98(3), 449-481.
Spillane, J. P., &: Thompson, C. L. (1997). Reconstructing conception of local capacity: The local education agency's capacity for ambitious instructional reform. Educational Evaluation and Policy Analysis, 19(2), 185-203.
Stiles, K., Loucks-Horsley, S., & Hewson, P. (1996, May). Principles of effective professional development for mathematics and science education: A synthesis of standards (NISE Brief, Vol. 1). Madison, WI: National Institutes for Science Education.
U.S. Department of Education, National Center for Educational Statistics. (1998). Toward better teaching: Professional development in 1993-94 (NCES 98-320). Washington, DC: Author.
U.S. Department of Education, National Center for Education Statistics. (1999). Teacher quality: A report on the preparation and qualifications of public school teachers (NCES 1999-080). Washington, DC: Author.
Webb, N. L. (1998). Criteria for alignment of expectations and professional development in mathematics and science education. Paper presented at the American Educational Research Association, San Diego.
LAURA DESIMONE is assistant professor of public policy and education at Vanderbilt University. Her specializations are education reform and program evaluation. She can be reached at email@example.com.
ANDREW C. PORTER is director of the Wisconsin Center for Education Research and professor of educational psychology at the University of Wisconsin, Madison. He can be reached at firstname.lastname@example.org.
BEATRICE F. BIRMAN is managing research scientist at the American Institutes for Research. Her specializations are education reform and program evaluation. She can be reached at email@example.com.
MICHAEL S. CARET is chief research scientist at the American Institutes for Research. His specializations are quantitative research methods, educational organizations, and education reform. He can be reached at firstname.lastname@example.org.
KWANG SUK YOON is senior research scientist at the American Institutes for Research. His specializations are motivation and lifelong learning, longitudinal research methods, and program evaluation. He can be reached at email@example.com.
i We estimate that 7% of districts nationally did not participate in the Eisenhower program at the time of our study. Thus, although not a national probability sample of all districts, our results can be taken as representative of 93% of the nation's school districts.
ii This report describes the Eisenhower Professional Development Program as it operated after the 1994 reauthorization and before the reauthorization that is expected to occur in late 2001 or in 2002. As this study was being completed, Congress was considering changes in the program as part of the reauthorization of the Elementary and Secondary Education Act.
iii This sample is representative of all large, urban districts, and most midsize districts. See "Data Sources" for more details on the sample.
iv The research literature contains a mix of large- and small-scale studies, including intensive case studies of classroom teaching, evaluations of programs designed to improve teaching and learning, and surveys of teachers about their preservice and in-service preparation and in-service professional development experiences. In addition, there is a large amount of literature describing best practices in professional development, drawing on expert experiences.
v There are other factors related to the quality of professional development that we do not measure (e.g., extent to which it matches the teachers' learning style, extent to which the professional development leader uses effective pedagogical strategies), so although the features in our analysis are associated with high-quality professional development, an activity having these characteristics might have other undesirable characteristics.
vi The type of activity was related to the core features primarily through duration, in that reform activities tended to be of longer duration than traditional activities. See Garet et al., 1999, 2002, for more details on these findings.
vii Despite the size of the literature on professional development, relatively little systematic, research has been conducted on the effects of professional development on improvements in teaching or on student outcomes, and very little has been conducted on the relative effects of alternative forms of professional development. The research that has been conducted, however, along with the experience of expert practitioners, does provide some preliminary guidance about the characteristics of high-quality professional development (see, in particular, Loucks-Horsley, Hewson, Love, & Stiles, 1998). In particular, several recent studies suggest that professional development that focuses on specific mathematics and science content and the ways students learn such content is especially helpful (Cohen & Hill, 2000; Kennedy, 1998).
viii Some districts who refused were replaced by other districts.
ix See Garet et al. (1999, 2002) for more details about sample design and selection.
x To identify consortium status, we asked each sampled district whether or not the district participated in the Eisenhower program through a consortium. If the district indicated that the district participated through a consortium, we then drew the entire consortium into our sample and adjusted the probability of the consortium being selected into the sample, based on the full set of member districts. In reporting results, we use "district" to indicate district or consortium, unless otherwise noted.
xi Estimating this model with structural equations (e.g., LISREL) would not enable us to estimate a non-recursive model, but would allow us to take into account the reliability of the measures (Bollen, 1989; Hanushek & Jackson, 1977).
xii The value of the scale for each district is based on the degree to which the district reports being aligned with whatever state and district standards and assessments exist in the district.
xiii The scale in theory runs from 0 to 14, but the maximum value in our sample was 10. That is, no district had all 14 programs, used all 14 to support professional development, and cofunded with all 14.
xiv In school districts, decisions about professional development may be made at some level in between the district and the school, by clusters of schools. These clusters may be "feeder patterns" (e.g., a high school and the elementary and middle schools that "feed" it), or they may be some other group of schools (e.g., all high schools or all elementary schools in the district).
xv Our measures of quality were limited in that on the district survey, we did not ask questions about the content-knowledge focus of the activity, and we did not ask about coherence because it seemed unlikely that district coordinators would have accurate information about these features, for example, the extent to which one particular activity was consistent with teachers' other experiences.
xvi As we explained earlier, the probability of a district being selected into our national sample was proportional to the number of teachers in the district. Consequently, all of the results are weighted by the size of the district (i.e., the number of teachers in the district). As a result, our data reflect information according to the percent of teachers in a district. For questions that ask about teacher participation in activities supported in full or in part with Eisenhower funds, or Eisenhower-assisted activities, we report the number of participations rather than the number of participants. As a rule, districts are unable to determine whether Eisenhower participants attended multiple Eisenhower-assisted activities. Therefore, a single participant may account for more than one "participation."
xvii We did not ask coordinators the number of contact hours that teachers spent in reform activities, since this information seemed inappropriate or inapplicable for most of the activities (e.g., participation in teacher networks if often ongoing).
xviii The five types of active learning are (1) observe leaders demonstrate; (2) observe other teachers; (3) simulate practice; (4) discuss problems with the implementation of new techniques; and (5) have their own practice observed.
xix High poverty was defined by 50% or more students eligible for free or reduced-price lunch.
xx Where we use the term districts in reporting our results, we refer to the national probability sample of Eisenhower-supported districts in our study.
xxi Although these data provide information about whether or not district coordinators engage in particular types of support and guidance, we have no information on the quality or frequency of these activities.
xxii The term Eisenhower-assisted indicates that an activity is supported in full or in part by Eisenhower funds.
xxiii We expect that the same increase in district size has a smaller effect on implementation and outcome factors as size increases. For example, a 100-teacher increase in district size from 100 to 200 teachers would have more of an effect on implementation and outcome factors than a 100 teacher increase in district size from 1,000 to 1,100 teachers. Taking the log of district size allows measured changes in the dependent variable to be associated with proportional increases in the number of teachers in a district.
xxiv The effect of size may differ for consortia and individual districts. Measuring the interaction of the log of district size and consortium status allows us to take this into account.
xxv b represents the standardized beta coefficient, or the standardized regression coefficient, which indicates the strength of the relationship between the two variables. For example, the beta of .17 for the relationship between cofunding and targeting means that for every one standard deviation increase in cofunding, there is a .17 SD increase in targeting. The arrow from cofunding to targeting indicates that targeting was regressed on cofunding.
xxvi For results from the case studies, see Garet et al., 1999.
xxvii As noted earlier, data for districts represent teachers in districts; for ease of presentation, however, here we refer to "districts."
xxviii Case study data also support this finding that high-quality activities can be planned at either the district or school level, and, likewise, low-quality activities can be planned at either level.
xxix Our case data illustrate that the level at which planning occurs, and teachers' involvement in it, are not necessarily related to the occurrence of professional development at the school itself, who participates, or the quality of the activities.
xxx Our case studies showed, however, that some small districts provide high-quality professional development to their teachers.