Teacher Participation in Content-Focused Professional Development & The Role of State Policy
by Kristie J. R. Phillips, Laura M. Desimone & Thomas M. Smith - 2011
Background/Context: Recent research has demonstrated the potential for teacher professional development to enhance teacher learning, improve instruction, and increase student achievement. Nevertheless, research examining the relationship between state and local policies and teachers’ participation in professional development is sparse. This connection between policy environments and teacher-based outcomes becomes increasingly important as educational reforms place new demands on teachers. Since professional development is a key mechanism to improving teachers’ instruction and students’ achievement, we address the extent to which state and school policy environments are associated with teachers’ participation in content-focused professional development. We consider such policy environments within the context of both mathematics, a high-stakes subject area, and science, currently a low-stakes subject area.
Purpose/Objective/Research Question/Focus of Study: In describing state policy environments along several dimensions, we seek to discover which types of policies are more or less influential in moving teachers into the types of professional development that research has shown to be most effective for improved teaching and learning.
Research Design: Using a national sample of high school mathematics and science teachers from the Schools and Staffing Survey (SASS), we conduct a secondary analysis using a three-level hierarchical linear model (HLM) to predict teachers’ level of participation in different types of professional development. We conduct separate analyses for mathematics (a high-stakes subject area) and for science (currently a low-stakes subject area).
Findings/Results: We find that the policy context at both the school- and state-level is more predictive of teacher participation in effective professional development in a high-stakes subject (mathematics) than a low-stakes subject (science). We also find that the alignment between state standards and assessments is a key attribute of state-level policies that tend to promote teacher participation in content-focused professional development in high-stakes subject areas. Even though state-level policies are important in promoting participation in effective professional development, we find that policy environments are strongest when they are closest to the teacher.
Conclusions/Recommendations: We conclude that both state- and school-level policy environments are associated with teachers taking high-quality professional development, but these findings are most pronounced in high-stakes subject areas. We also find that policies promoting consistency in the form of alignment between standards and assessments are perhaps the most important type of policies that states can adopt to encourage teachers to participate in effective professional development.
Research has consistently demonstrated the importance of effective professional development for teacher learning and for improved instructional practicesboth of which are related to student achievement. In response to standards-based reforms such as the No Child Left Behind Act (NCLB) and their emphasis on teacher quality and accountability, states have adopted educational policies aimed at improving teaching through professional development. However, not all professional development is equally effective in helping teachers improve their skills. Thus, policy makers face the challenge of adopting policies that not only promote professional development for teachers but policies that encourage teachers to participate in types of professional development that have been demonstrated to improve teacher practices and subsequently, student achievement. Nevertheless, even in an educational era shaped by standards-based reform policies that emphasize teacher quality, little is known about how policy environments shape teachers participation in professional development.
In this study, we consider the emerging prominence of accountability and other policy mechanisms as we address the extent to which state and school policies are associated with teachers participation in professional development. We examine the influence of these policies within the context of both mathematics, a high-stakes subject area, and science, currently a low-stakes subject area. In describing state policy environments along several dimensions, we seek to discover which types of policies are associated with moving teachers into the types of professional development that research has shown to be most effective for improved teaching and learning.
IMPORTANCE OF PROFESSIONAL DEVELOPMENT
Current educational policies in the U.S. rely on standards-based reforms to improve teaching and learning. These reforms include the setting of high standards, the development of curriculum frameworks, and the alignment of standards and curricula to assessments. Such reforms, including those outlined in NCLB, have generated new expectations for teachers and the achievement of their students (Bybee, 1993; National Council of Teachers of Mathematics, 1991; National Research Council, 1996; Webb & Romberg, 1994). These new expectations require that teachers develop a deep understanding of the content they teach (Ma, 1999). Even though teachers generally support the implementation of high standards for teaching and learning, many teachers do not have the necessary content knowledge of the subjects they teach that would aid them in meeting these high standards (Cohen, 1990; Elmore, Peterson, & McCarthey, 1996; Grant, Petersen, & Shojgreen-Downer, 1996; Sizer, 1992). Research has indicated that, on average, both pre-service and in-service teachers have a weak content knowledge of the subjects they teach (e.g., Ball, 1990; National Commission on Teaching & Americas Future [NCTAF], 1996; Rech, Hartzell, & Stephens, 1993; Tirosh & Graeber, 1989). As such, content-focused, in-service professional development for teachers is a cornerstone of standards-based reform.
CONTENT-FOCUSED PROFESSIONAL DEVELOPMENT
Research has identified five major characteristics of professional development associated with improving teachers knowledge, skills, and teaching practices. These characteristics include the following: (1) a focus on subject matter content and how students learn that content; (2) opportunities for active learning; (3) coherence with teachers other professional activities and experiences; (4) involvement with teachers from the same school, grade, or department to allow for the sharing of ideas; and (5) a substantial duration of contact hours that are sustained throughout the year (e.g., Desimone, Porter, Birman, Garet, & Suk Yoon, 2002; Garet, Birman, Porter, Yoon, & Desimone, 2001; McLaughlin & Talbert, 2001). These attributes of professional development have also been linked to student achievement (Cohen & Hill, 2000; Kennedy, 1998). As a result, recent standards-based reforms (including NCLB) have focused on the importance of improving the quality of teaching by increasing teacher participation in the types of professional development that feature these qualities. This is in contrast to the much-maligned but ever-resilient and still prevalent one-shot workshop (Garet et al., 2001), which often focuses on management, discipline, or administrative issues rather than on subject matter content.
Of all these features associated with high quality professional development, the focus on subject-matter content has the strongest relationship with student achievement (Cohen & Hill, 2000; Kennedy, 1998; Wenglinsky, 2000, 2002) and seems to hold the most promise for fostering real change in teachers knowledge and, subsequently, in their instruction and in their students learning (Desimone et al., 2002; Garet et al., 2001). Furthermore, research indicates that professional development that focuses on specific content and the ways students learn such content is particularly helpful in improving students conceptual understanding of the subject matter (Cohen & Hill, 1998). As a result, sustained, content-focus professional development has emerged as perhaps the most important type of in-service teacher education.
Research has also demonstrated that very few teachers actually participate in content-focused professional development (Desimone, Smith, & Ueno, 2006; Garet et al., 2001). One reason for this is that teachers generally self-select into types of professional development. Although states have requirements for continuing education and in-service professional development hours, the actual content of the professional activities is usually the teachers choice. This was demonstrated in a national study that found that nearly 70% of teachers nationwide choose their own professional development activities (Garet et al., 2001). Because content-focused professional development is associated with improved teaching and improved student learning, this finding leads us to question which state and school policies might encourage teachers to self-select into content-focused professional development. Such policies, if developed effectively, could lead to increased knowledge and skills in the existing teaching force.
POLICY & PROFESSIONAL DEVELOPMENT
Because improvements in teaching and learning rely heavily on teachers experiences in professional development (Corcoran, Shields, & Zucker, 1998; Loucks-Horsley, Hewson, Love, & Stiles, 1998; NCTAF, 1996; Sykes, 1996), and because content-focused professional development is associated with improved teaching and learning (Cohen & Hill, 2000; Desimone et al., 2002; Garet et al., 2001; Kennedy, 1998; Wenglinsky, 2000, 2002), it is critical to understand which policies work best to foster teachers participation in content-focused professional development. Previous research has identified district policies that are associated with increased participation in content-focused professional development, such as teacher participation in planning the professional development, the districts engagement in continuous improvement efforts, the alignment of the professional development with other reform efforts in the district, and using multiple sources to co-fund the activities (Desimone, Garet, Birman, Porter, & Yoon, 2001). Similarly, research has identified several policies that are related to the provision of higher-quality professional development: (1) the alignment of standards and assessments to professional development, (2) continuous improvement efforts, and (3) coordination between postsecondary institutions and school districts (Desimone, Birman, Porter, Garet, & Yoon, 2003). However, we know little about the analogous roles of state and school policy in moving teachers toward content-focused professional development.
Furthermore, there is a question as to how education policies influence teachers and administrators behavior in the context of high-stakes accountability. That is, NCLBs high-stakes testing requirements have encouraged districts and schools to emphasize tested subjects and the skills and content associated with those subjects. Specifically, NCLB has increased the time and emphasis on mathematics and English language arts and decreased time and emphasis spent on other subjects (Center on Education Policy, 2007). To account for the influence of high-stakes testing requirements, we explore the role of state and school policies in mathematics, a high-stakes subject area, and in science, currently a low-stakes subject area.
POLICY ATTRIBUTES THEORY
Our analysis of the potential for state and school policies in promoting effective professional development for teachers is grounded in the policy attributes theory. This theory is rooted in Webers (1947) classical theories on authority, social action, and rationalization. The policy attributes theory, developed by Porter and colleagues (Porter, 1994; Porter & Brophy, 1988; Porter, Kirst, Osthoff, Smithson, & Schneider, 1993; Schwille et al., 1988), identifies five attributes that contribute to the successful implementation of a policy: consistency, specificity, authority, power, and stability. Consistency refers to the extent to which all components of a system are aligned with one another. Specificity identifies the extent to which clear and detailed guidelines are specified. Authority refers to the degree to which a policy is supported and legitimized by relevant individuals or institutions. Power is a measure of the rewards and sanctions attached to a policy. And finally, stability identifies the extent to which policies and practices remain in place over time.
The policy attributes theory works as a framework for identifying and analyzing the policies that states, districts, and schools use to implement reforms (Desimone, 2002a). It provides an analytical foundation from which researchers have drawn insights about the influence of policies and reforms in several policy contexts. Specifically, it has been used to study systemic reform (Clune, 1998), comprehensive school reform (Berends, Chun, Schuyler, Stockly, & Briggs, 2002; Desimone, 2002a), teacher quality (Desimone, Smith, & Frisvold, 2007), effects of NCLB on achievement (Desimone, Smith, Hayes, & Frisvold, 2005), effects of school policies on teacher professional development (Desimone, Smith, & Phillips, 2007), and teacher decision making in general (Porter, Archbald, & Tyree, 1990; Porter, Floden, Freeman, Schmidt, & Schwille, 1988). It also applies to this study in that it provides an organizing framework for describing the way state and local policies influence teacher participation in professional development activities.
POLICY ATTRIBUTES & EDUCATIONAL REFORM
The link between state policy and improved instruction (both directly and indirectly, as state policies are filtered through districts and schools) has been well documented (Bryk, Sebring, Kerbow, Rollow, & Easton, 1998; Elmore, 1993; Fullan, 2001; Hannaway & Kimball, 1997; Murphy & Hallinger, 1992; Purkey & Smith, 1983; Sebring & Bryk, 2000). Nevertheless, little is known about the specific characteristics of education policies that most successfully lead to improvements in teaching and learning (e.g., Spillane & Louis, 2002). We briefly describe the extant literature to explain the importance of each of the five policy attributes and how they relate to policies at both the state and school levels. We also identify how these policies relate to standards-based reform, as well as how they are expected to be associated with improved teaching.
Consistency is the establishment of coherent policies and programs that build upon one another and align with each other. At the state level, when policies within a state are consistent with one another, each educational strategy supports and reinforces the others (Clune, 1998; Doolittle, Herlihy, & Snipes, 2002; Porter & Brophy, 1988; Smith & ODay, 1991). In the current policy environment of NCLB, one important dimension of consistency would ideally be the alignment of state standards with assessments, curricula, and teacher professional development (Cohen, 1995; Corcoran, 1997; Schmidt & Prawat, 1999). At the school level, consistency indicates how well policies that directly affect incentives and opportunities for teachers are aligned with state-, district-, and school-level standards. For instance, such policies might include guidelines for the design and focus of professional development that are available to teachers in a particular district or school. Such policies could either promote alignment or create inconsistencies in the expectations for teachers at various levels in the educational hierarchy. For example, a common criticism from teachers is that a new reform they are asked to follow contradicts a previous reformor worse yet, a concurrent reform (Datnow & Stringfield, 2000; Ross, Alberg, & Nunnery, 1999). Alternatively, when reforms at all levels (state, district, and school) are aligned with each other and are pushing teachers in the same direction (i.e., when they are consistent), they are more easily and readily implemented (Newmann, Smith, Allensworth, & Bryk, 2001).
Specificity refers to the extent to which a policy provides clear and detailed guidance as to what schools and teachers are to do. In the case of standards-based reform, standards often lack specificity (Porter, 1994); therefore, translating them into practice leaves a great deal of room for interpretation when specific guidelines and strategies are not provided (Chatterji, 2002; Porter & Smithson, 2001). At the state level, policies exhibit specificity when they are accompanied by clear, detailed guidelines and strategies for implementing the standards set forth by the state (Chatterji, 2002; Porter, 1994; Porter & Smithson, 2001). Specificity is often evidenced in the clarity of state standards, and whether they are specified for each grade instead of for grade bands (e.g., having one set of standards for Grades 3-5). At the school level, specificity can be manifested in the detailed guidance provided to teachers, for example, whether they are provided with a particular curriculum and pacing sequence, with a general philosophy of instruction, or with a target that leaves the achievement strategies up to the teacher. Additionally, professional development can be a powerful force in providing specifics about a particular reform that results in teachers changing their behavior to accommodate the reform (e.g., Cohen & Hill, 2000; Cook, Murphy, & Hunt, 2000; Desimone, 2002).
Another attribute of an effective policy, authority, represents the degree and type of backing and support a policy engenders. Policies can have authority through laws, through social or professional norms, and through the support of influential institutions or individuals (Porter, 1994; Porter & Brophy, 1988). Authority can also be established through the degree to which fiscal support offered in the form of resources backs the policy. Funding attached to a policy often signals the importance policy makers place on it, which can influence buy-in (Desimone, Smith, Hayes, & Frisvold, 2005; Smith, 2006). For example, a state-level policy on professional development could have authority through requirements that mandate teacher participation in professional development for a certain number of hours. At the school level, a policy usually has authority when it enjoys the buy-in of the principal and teachers who are charged with implementing it. Authority can be realized when a policy becomes part of teacher or school norms, or has the backing of a well-respected institution (e.g., the American Federation of Teachers) or an individual (e.g., a well-liked principal or district superintendent). Authority can also be achieved through the participation of implementers in the design of the policy. For example, when teachers play a role in shaping the policies they are supposed to implement, the policy may become persuasive to teachers through their own involvement and buy-in (Spillane & Jennings, 1997). Such buy-in has long been noted as a key component of teachers adoption and implementation of reform efforts (e.g., Datnow, 2000; Desimone, 2002a; Tyack & Tobin, 1994).
A policy is considered to have more power behind it when consequences are attached to it (Porter & Brophy, 1988). When states offer rewards or implement sanctions in response to schools or districts meeting certain policy-related benchmarks, states exhibit power. Such pressures to respond to certain policies are characteristic of standards-based reforms (Herman & Golan, 1991; Massell, 1998, 2001; Romberg, Zarinnia, & Williams 1989), including NCLB. The NCLB power mechanisms include the evaluation of schools based on student achievement scores and the evaluation of teachers based on their credentials. If teachers follow a policy because of its power, they implement it only because of the threat of sanctions or the desire to earn a reward. If, however, they follow a policy because of its authority, they implement it because they have been persuaded that it will benefit them or their students. Both authority and power have demonstrated a relationship to policy implementation (e.g., Berends, Bodilly, & Kirby, 2002; Berman & McLaughlin, 1975; Datnow, 2000; Louis & Marks, 1998). However, evidence suggests that power sometimes results in shallow, short-term implementation, whereas authority is more likely to result in longer term and deeper implementation (Desimone, 2002a). At the school level, indicators of power could include the degree to which principals observe and evaluate teachers, and the degree to which it is possible for teachers to be dismissed on the basis of their performance (see Desimone, Smith, & Phillips, 2007).
The last policy attribute that we consider is the stability of the policy environment. How long policies and people remain a stable part of the policy landscape has a significant influence on the level and quality of implementation (Huberman & Miles, 1984). Studies of standards-based reform in particular indicate the need to counteract the forces of political instability that cause shifts from one policy directive to another (Chatterji, 2002; Cohen, 1996; Elmore, Abelman, & Fuhrman, 1996). While stability is not as readily manipulated from a policy perspective as the other policy attributes (e.g., educational leaders can attach a reward to a particular policy, but legislating stability is not as straightforward), it plays an influential role in the success of any educational reform effort (e.g., Mirel, 1994; Timar, 1989). While stability is difficult to define at the state level, it becomes very important at the school level. Conditions such as high teacher and administrator turnoverparticularly in lower income communitiesoften make policy stability difficult to achieve. Additionally, most teachers understand that reforms come and go. As a result, they often greet a new policy with a wait and see attitude, expecting that it will disappear over the horizon like so many of its predecessors (Ross et al., 1997). The longer a policy remains in place, or the longer a principal, district superintendent, or teachers remain in their jobs, the more stable the policy environment. This type of stability provides a supportive context for policy implementation (Berends, Chun, et al., 2002).
This paper focuses on how state and school policies shape teachers participation in professional development, which we see as reflective of both teachers decisions about which types of professional development to take, and districts and schools decisions about which types of professional development to offer. Of specific interest are the policies that promote teacher participation in content-focused professional development, which, according to the literature described earlier, is a critical factor in improving teaching and learning. Specifically, we ask the following research questions:
To what extent are state-level policy environments associated with the number of hours teachers spend participating in professional development focused on subject matter content, rather than on instructional strategies, classroom management, or no professional development at all?
How important are state policies in promoting professional development relative to more localized school policies?
Is the relationship between state and school policy and professional development participation stronger for a high-stakes subject (e.g., mathematics) than for a lower-stakes subject (e.g., science)?
Research has shown weak links between accountability and student outcomes (e.g., Carnoy & Loeb, 2003; Hanushek & Raymond, 2004); however, we have found no research that has systematically examined whether or not the policies associated with educational accountability promote teacher engagement in professional development activities that have been shown to improve teaching and learning. To address this issue, we intend to accomplish the following in our study: First, we provide a comparative examination of the relationship between different policy attributes at both the state and school levels and their associations with teachers participation in professional development activities; and second, we explore whether the pressure of teaching a high-stakes subject makes these possible associations between policy and teacher participation in content-focused professional development any stronger.
In this study, we assess the associations between policy and teachers participation in content-focused professional development. We hypothesize that state and school policies have the potential to move teachers away from professional development focused on topics such as classroom management and discipline and toward content-focused professional development. Furthermore, the degree to which a policy has more of each policy attribute, the more we expect the policy to influence practice. Further, we expect state policy attributes to behave differently than school policy attributes. State policies are often filtered through district and school policies; therefore, we predict that the policy environment carries more influence when it is closest to the teacherespecially when predicting teacher behaviors such as participation in professional development.
We also hypothesize that, in high-stakes subjects such as mathematics, the attributes of policies (both at the state and school levels) will have a stronger relationship with teachers taking content-focused professional development than in lower-stakes subjects such as science, a subject that is not yet included in the calculation of whether a school is failing under NCLB. If the policy attributes work as indicated by the theory, which suggests that more of each improves implementation, we expect a high-stakes accountability environment to enhance the influence of policy attributes in moving teachers into content-focused professional development. For example, a mathematics teacher is likely to feel pressure to increase her students mathematics test scores because of the high-stakes accountability environment associated with mathematics. This pressure would support existing policies designed to encourage teachers to participate in professional development. Thus, policies offering rewards and sanctions attached to test scores might play more of a role in moving the teacher into content-focused professional development, when compared with a subject area in which there is less external pressure, such as science. Therefore, we expect that attributes of policies would be more influential on teacher professional development participation in high-stakes subjects. In subjects such as science, in which external pressures are not as great, we would not expect the relationship between policy attributes and teacher participation in content-focused professional development to be as strong.
DATA & METHODS
This study uses data from the Schools and Staffing Survey (SASS), the nations most extensive survey of elementary and secondary schools and the teachers and administrators who staff them. It is a nationally representative sample of teachers and schools and includes a random sample of schools stratified by state, public/private sector, and school level. The survey examines issues such as teacher demand and shortage, teacher and administrator characteristics, school programs, and general conditions in schools (U.S. Department of Education, 2002). While the SASS was not specifically designed to evaluate causal relationships between professional development participation and state- and school-level policies, it provides sufficient information to examine trends and relationships between the types of professional development teachers participate in as well as state and school policies. Furthermore, it is the only available dataset through which we can identify systematic relationships between professional development and state and school policies on a large, nation-wide scale. We rely on theory to guide the interpretations of our results, and we suggest that the work we have done here is useful for enabling a broader look at policy systems and examining whether theories about the relationships between policy and teacher participation in professional development emerge with national data.
Our analyses included the restricted-use version of the 1999-2000 SASS public school survey. We use both the administrator questionnaires and the questionnaires administered to a linked random sample of teachers within each school. While the total 1999-2000 SASS sample was comprised of about 52,000 elementary and secondary teachers, our analyses focus on public high school teachers whose main assignment fields were either math or science, as well as their principals. Because our dependent variable is a composite measure indicating the number of hours teachers participate in content-focused professional development, we restrict our analysis to high school teachers whose main assignment fields were either math or science.1
Analyses for math and science teachers were conducted separately; therefore, we report two different sets of resultsone set for mathematics and one set for science. Our analytic sample of high school teachers who reported mathematics as their main assignment field was 2,008, and 1,819 high school teachers who reported science as their main assignment field. Sample weights were used to compensate for the over- and under-sampling of schools and teachers in the complex stratified survey design. Each teacher and administrator was weighted by the inverse of the probability of their selection in order to obtain unbiased estimates of the national population of public schools and teachers in the year of the survey. Because SASS is a nationally representative sample of teachers, it offers us the opportunity to explore our hypotheses in a broad sense by linking principals, teachers, and the policy environment in which they work.
Our analyses measure the relationship between teachers participation in professional development and attributes of the state and school policy environments. We controlled for teacher and school characteristics that were likely to be related to the policy and professional development variables. Each of the measures used in our analyses are described below. Furthermore, we provide an appendix that outlines the exact SASS questions that correspond to each of the teacher, principal, and school variables, which describes how we coded the variables, and also provides the means and standard deviations (where applicable) for each of the two sampleshigh school math teachers and high school science teachers (see Appendix A). This appendix also includes a description of each of the state-level policy measures we use and how they are operationalized.
Dependant variables. Professional development takes many different forms, ranging widely in terms of quality and effectiveness. We focused on what research has shown to be among the most salient characteristics for effects on teaching and learninga focus on subject-matter content (Cohen & Hill, 2000; Desimone et al, 2002; Garet et al., 2001; Kennedy, 1998; Louis, Marks, & Kruse, 1996). We contrast the number of hours a teacher spends in content-focused professional development with the number of hours spent on non-content or instruction related professional development activities that cover topics such as instructional strategies and classroom management. We do not imply that professional development intended to help teachers manage the classroom is not useful or important. Rather, given our focus on policy, we explore the associations of policy with participation in content-focused professional development, a type of professional development that is most likely to influence instruction.
We acknowledge that our measure of content-focused professional development participation is not without weaknesses. A perfect measure of quality for any professional development activity would include data on all of the components that the literature suggests are related to changes in teaching and learning. Ideally, we would like to know the full range of characteristics that define a teachers experiences in professional development, including span of time, contact hours, alignment of professional development activities with curriculum, policies, or educational initiatives, and whether teachers participated with other teachers from their school or grade. However, such data are not available in SASS, or any other nationally representative dataset. Nevertheless, we have confidence in the measures of quality we have chosen for this study, since focus on content is consistently found to be a leading feature of quality professional development (e.g., Cohen & Hill, 2000; Garet et al., 2001; Kennedy, 1998), and classroom management/ discipline has also consistently shown to be unrelated to improved teaching and student achievement (e.g., Loucks-Horsley et al., 1998).
Because we assume that participation in one type of professional development was likely to be conditional upon how much professional development of other types was taken during the school year, we predict total hours of participation in professional development as a function of the type of professional development participated in: 1) content-focused professional development, 2) professional development focused on teaching strategies, and 3) professional development focused on classroom management. We first calculated the total hours each teacher spent in professional development, and then accounted for the hours spent in each of the three types of professional development we identify in this study. Total hours of participation in professional development were measured by the number of hours teachers reported participating in any type of professional development. Because the SASS questions about professional development asked teachers to identify duration categories that described their participation in various types of professional development (and not the exact number of hours they spent in professional development), we recoded responses to represent the midpoints of each duration category. Possible responses about the number of hours teachers spent taking professional development were 0 = did not participate; 4 = 8 hours or less; 12.5 = 9-16 hours; 24.5 = 17-32 hours; and 40 = 33 hours or more. These recoded responses enabled the creation of a normally distributed, ordinal measure of the number of hours each teacher spent participating in professional development.
To account for each of the three types of professional development teachers participated in, we included dummy variables for each type: content-focused, teaching strategies, and classroom management. In our analyses, each of these three dummy variables were interacted with the total number of hours each teacher spent in professional development, which created an indicator measuring the number of hours a teacher spent in each type of professional development, relative to their participation in other types of professional development. In creating these measures, we considered professional development to be content-focused when it was centered on in-depth study of the content in a teachers main assignment field (either mathematics or science) and when it focused on the content and performance standards in a teachers main assignment field (in our study, either mathematics or science). Teaching strategies professional development in this study included professional development that focused on methods of teaching, use of computers for instruction, and methods of assessment. Classroom management professional development included activities that focused on student discipline and management in the classroom. For more detailed information on each of these measures, please refer to Appendix A.
Teacher background characteristics. We included control variables in our models to account for the teacher characteristics that previous research has shown to be related to teaching behaviors. We expected full-time teachers to invest more heavily in their teaching than part-time teachers; similarly, we expected teachers in mid-career and early career to be more active in seeking learning opportunities and welcoming of new reforms than teachers in the later stages of their careers (Berends, 2000). Further, we expected that teachers with advanced subject-matter degrees would be more likely than their colleagues without such degrees to feel comfortable seeking out more content-related professional development (Desimone & Smith, 2006). We also expected that teachers teaching an advanced class would be more likely to take professional development focused on content and/or instruction to meet the needs of their advanced students (Desimone, Smith, & Phillips, 2007). And finally, we expected that teachers without full certification would be more likely to take professional development to fulfill their certification requirements.
Motivated by research on teachers and also our hypotheses about which teachers would be most likely to take content-focused professional development, our analyses included controls for the following: whether or not a teacher was a regular, full-time teacher (as opposed to a part-time teacher); a teachers total years of experience (grand-mean centered); years of experience squared (grand-mean centered); teachers education level; whether or not the teacher taught an advanced class in mathematics or science; and teachers certification level (full, partial, or no certification). Teachers level of education was measured not only in terms of the highest degree achieved, but also in terms of the content focus of their degree. Specifically, our categories for math teachers were as follows: bachelors degree or beyond in math (reference category); bachelors degree or beyond in math education; minor in math or bachelors degree or beyond in science (a related subject); and finally, no major or minor in math or science. Similarly, our categories for science teachers were bachelors degree or beyond in science (reference category); bachelors degree or beyond in science education; minor in science or bachelors degree or beyond in math (a related subject); and finally, no major or minor in science or math.
Teaching advanced classes was measured differently for math and science. If teachers taught at least one class in advanced algebra, analytic geometry, pre-calculus, or calculus, they were considered to be teaching at least one advanced class in math (as opposed to those teachers who did not teach any of these classes). However, because it is less clear which science classes should be considered advanced from the categories listed in the SASS, we created three dummy variables to better understand the relationship between teaching advanced science classes and science teachers participation in professional development. The three categories included whether or not teachers taught at least one class in physics, chemistry, or biology. Teachers of other types of science classes were used as the reference category. We expected the physics and chemistry teachers to represent advanced science classes, but given that biology is offered at many different levels in high school curriculums (U.S. Department of Education, 1999), we did not necessarily consider it as a proxy for advanced science class, but rather treated it as a way of distinguishing different types of science teachers.
School characteristics. We expected more professional development participation in high-poverty districts than in low-poverty districts, due to the proliferation of programs and federal government funding (Elmore, 1993). Similarly, we expected more participation in urban than suburban or rural districts, due to the higher concentration of teachers and more specialized programs (Hannaway & Kimball, 1997). Thus, we included a control for whether the school is urban, rural, or suburban. We also controlled for the level of poverty of the student population, measured by the percentage of students who were eligible for free and reduced lunch (grand-mean centered). Missing values for school poverty were imputed using STATA statistical software, based on regression-based estimates using values of 17 items dealing with school climate. An imputation flag was created and included in all statistical models.
Policy attributes. Because the literature on school and state policies indicates that such policies do not act in isolation (Desimone 2002b; Hannaway & Kimball, 1997; Spillane, 1996), we addressed two levels of policies: the attributes of school-level policies and the attributes of state-level policies. Ideally, we would have examined district policy as well; however, since schools translate district policy and there is substantial variation between schools in the impact and implementation of district policy (Desimone, 2006), our examination of school policy in the context of state policy is a useful step toward understanding the complex interactions of multiple levels of educational policies.
SCHOOL LEVEL POLICY ATTRIBUTES
The 1999-2000 SASS teacher and administrator questionnaires contained items related to four policy attributes. From these items, we created one measure of consistency, two measures of authority, two measures of power, and two measures of stability.
Consistency. Our consistency measure intended to capture the degree of alignment between teachers professional development and other policies at multiple levels in the educational system. We measured the consistency of the policy environment with principal reports of how well the content of professional development for teachers was aligned with school-, district-, and state-level policies. We created a composite from nine items derived from two separate questions in SASS. Principals were asked, How important is each of the following in determining the in-service professional development activities of teachers in this school: (1) special state-level initiatives, (2) district-level initiatives or district improvement plan, (3) school improvement plan, (4) implementation of state or local ACADEMIC standards, (5) implementation of state or local SKILLS standards, and (6) teacher preference. Answers were reported in a range from 1 to 5, where 1 = not important at all and 5 = very important. These six items were combined with three additional items associated with the following question: How often is professional development for teachers at this school (1) designed or chosen to support the schools improvement goals, (2) designed or chosen to support the district's improvement goals, and (3) designed or chosen to support the implementation of state or local standards? (0 = never, 1 = rarely, 2 = sometimes, 3 = frequently, and 4 = always). These items were combined into one composite measure because the importance and frequency of these activities demonstrate the degree of alignment and coherence between policies, all of which correspond with characteristics of consistency. Factor analyses confirmed that each of these items load on a single construct, which indicated that the separate items all measure the same dimension or construct, and the composite indicator was created by summing the items (Cronbachs alpha = .80).2
Authority. Our measure of authority reflects the idea that policies are likely to have increased support and be persuasive to teachers when the teachers play an active role in shaping and influencing those policies. Our first authority measure included teacher reports of how much influence they thought teachers in their school had over school-level policies. We created a composite indicator of teacher influence over school policy by summing teachers responses, where 1 = no influence and 5 = a great deal of influence, to the following seven items: setting performance standards, establishing curriculum, determining the content of in-service professional development programs, evaluating teachers, hiring new full-time teachers, setting discipline policy, and deciding how the school budget will be spent. A composite measure was created by summing these items (Cronbachs alpha = .80). We grand-mean centered the variable, indicating that the variable measures the difference between an individual teachers perception of the level of influence teachers had over policy in their school, relative to the average math or science teacher sampled in the SASS.
Our second measure of authority reflects teacher reports of their level of control over classroom practices. We created a composite variable from six items that asked teachers how much control they had over planning the following in their classroom: selecting textbooks and other instructional materials; selecting content, topics, and skills to be taught; selecting teaching techniques; evaluating and grading students; disciplining students; and determining the amount of homework to be assigned (1 = no control and 5 = complete control). The composite indicator was created by summing the items (Cronbachs alpha = .77). As with the authority measure of control over school policy, we grand-mean centered the teacher influence over classroom policy indicator, which can be interpreted as a teachers difference from the average teacher influence over classroom policy score.
The third and final measure of authority describes principals reports of the degree to which teachers take leadership roles in designing and/or implementing professional development activities. The composite measure, teacher leadership in professional development, was created by summing across two items: administrator reports indicating (1) how often professional development is planned by teachers, and (2) how often it is presented by teachers in the school or district (0 = never, 1 = rarely, 2 = sometimes, 3 = frequently, and 4 = always). Cronbachs alpha for this composite was .70.
Power. In this study, our power variables were intended to capture sanctions that were associated with a schools policy environment. Each of our two measures of power came from principals reports of policies and practices at their school, and each of the measures was grand-mean centered. Our first measure, barriers to teacher dismissal, was a composite created as a sum of responses to a question that asked about the extent to which each of the following six items can be considered barriers to dismissing poor or incompetent teachers: (1) personnel policies, (2) termination decisions not upheld by third party adjudicators, (3) inadequate teacher assessment documentation, (4) tenure, (5) teacher associations and organizations, and (6) dismissal is too stressful and uncomfortable for those involved. Principals reported that each of these items either were (coded 1 = yes) or were not (coded 0 = no) barriers to dismissal in their school. The more barriers to firing teachers, the less powerful the local policy environment is likely to be. We would expect teachers to feel more safe and secure in their jobs, and less at risk for negative sanctions (such as being fired), if there were many barriers to the principal firing them. Our second measure of power was principals reports of how often they supervised and evaluated their faculty and other staff. Response categories were 1 = never, 2 = once or twice a month, 3 = once or twice a week, and 4 = every day. Without active monitoring, it would be difficult to reward or sanction instruction (unless only test scores were used); therefore, we consider this to be an appropriate measure of power.
Stability. According to the policy attributes theory, stability represents the extent to which people, circumstances, and policies remain constant over time. The SASS did not directly allow us to examine the stability of circumstances or policies, but we can, in specific ways, examine the stability of the schools labor force over time. Our two measures of stability included the frequency of principal and teacher turnover at schools. Our first measure was a dummy variable coded 1 if a principal had been at a school for three or more years and coded 0 if the principal had been there for less than three years. Our second measure was a percentage of teachers who participated in the SASS who had been at their current school for at least three years. Our stability of teachers variable is grand-mean centered in the analyses.
STATE LEVEL POLICY ATTRIBUTES
To develop state-level measures of the policy attributes for our analysis, we constructed a State Policy Database from existing national data sources. These sources include Education Weeks Quality Counts report (Education Week, 2000), the American Federation of Teachers report on states titled Making Standards Matter (American Federation of Teachers, 2001), and the State of State Standards report published by the Thomas B. Fordham Foundation (Finn & Petrilli, 2000). We also created measures from several SASS (2000) questions asked of public school principals that were related to the policy environment in their state. Principals responses to these items were aggregated to the state level (using the appropriate sample weights), creating a percentage of principals in each state who responded accordingly. All of our variables were measured in year 2000 to correspond with the data we used in the SASS. Descriptions of each of our measures of state policy attributes are described below.
Consistency. Our state-level consistency measure indicates whether states achievement tests are aligned with state standards. As reported by the Quality Counts report (Education Week, 2000), we use a dichotomous measure (0 = no and 1 = yes) of whether or not a state required a statewide test that was custom-developed to match state content standards, also known as a criterion-referenced test (CRT). To account for our subject-specific analyses in both math and science, we used two separate measures: one indicating that a test was custom-developed to match state content standards in mathematics and another for science.
Authority. At the state level, our measure of authority described the extent to which states require and fund professional development. Such a resource provision is an important form of support that contributes to establish a policys authority because it establishes backing and support for policies related to teachers and teaching. We used Education Weeks assessment of the degree to which professional development was supported by the state. The Quality Counts report 2000 rated states as a 0 if the state did not require professional development for teachers, as a 1 if the state required but did not finance professional development for teachers, and as a 2 if the state both required and financed professional development for teachers.
Power. We measured state-level power through an eight-item composite of the number of sanctions and accountability mechanisms available in a state. The power indicators were taken from the Quality Counts report (Education Week, 2000) as well as the State of State Standards report (Finn & Petrilli, 2000). The items used to create this composite measure were: (1) The state assigned ratings to all schools or identified low-performing schools; (2) Teacher evaluations were tied to student achievement; (3) State issued a report card on individual schools, including school test scores; (4) State required that school report cards be sent home; (5) State could impose sanctions on schools; (6) State had policies encouraging pay for performance; (7) State had a report card and disaggregated information; (8) School or district report cards included at least one piece of information related to teacher quality. A score of 1 indicated yes for each item used in the composite, and a score of 0 indicated no. The scores for each item were summed individually for each state and centered based on the nationwide average. This identified the power of the policies in each state relative to other states.
Specificity. Two state-level measures of specificity were also included in our analyses. The first measured the extent to which materials were made available for math and science at the state level, with the idea that such resources were used to provide detail and clarity about a particular policy. The American Federation for Teachers (2001) identified five areas where states can offer resources for instruction: (1) State identified instructional resources that were aligned to the standards in the field of math; (2) State provided information on instructional strategies or techniques to help teach the standards in the field of math; (3) State disseminated lesson plans and units based on standards in the field of math; (4) State provided performance indicators to clarify the quality of student work required for mastery of the content standards in the field of math; and (5) The math curriculum laid out the learning continuum that showed the progression and development of knowledge and skills from grade to grade. Each of these items were summed to create a Materials composite, where a score of 0 indicates that no materials were offered by the state; 1 means that one of the five types of materials were offered by the state; 2 means that the state offered two types of materials; 3 = three types of materials; 4 = four types of materials; and 5 = five out of the five possible types of materials were offered by the state. Again, this measure was subject-specific; therefore, we created separate composites indicating the level of materials available in mathematics and another addressing the level of materials available in science.
Our final measure of specificity was a Quality Counts (Education Week, 2000) rating of whether or not a state has clear and specific standards. It was a dichotomous measurethe state either had clear and specific standards (by Quality Counts standards), or it did not. This rating largely determined the clarity and specificity of state standards by whether or not each state described specific standards by grades and by individual subject areas. Like several other state-level measures, we included two measures of specificity: one that measured the clarity and specificity of state standards in high school mathematics and one specific to state standards in high school science.
Our analyses were conducted in two stages, first with our sample of high school mathematics teachers and then for the high school science teacher sample. This allowed us to examine whether or not results were similar for a high-stakes subject (mathematics) and a low-stakes subject (science). For both stages of these analyses, we used a 3-level hierarchical linear model (HLM) to predict teachers level of participation in different types of professional development activities. The Level 1 portion of the models included all three types of professional development teachers participated in as a function of their total hours spent in professional development. Teacher background characteristics and school characteristics (including school-level policies) made up the Level 2 portion of the models, and state policy measures were included at Level 3.3 The models for math and science used the same variables, with the exception of two teacher background characteristics: teacher education level and whether or not a teacher teaches advanced classes. Both of these variables were constructed to account for subject-specific nuances in teacher education and patterns of teachers course teaching; therefore, changes to these respective variables were made to more accurately accommodate math- and science-specific models.
We predicted hours of participation in professional development as a function of the type of professional development participated in: 1) content-focused professional development, 2) professional development focused on teaching strategies, and 3) professional development focused on classroom management. While we could have modeled each of these as separate dependent variables, we would have been making the implicit assumption that the taking of one kind of professional development was unrelated (i.e., uncorrelated) with the taking of other kinds of professional development. It is more likely, however, that decisions regarding participation in one type of professional development are conditional on how much professional development of other types has either been taken or is anticipated to be taken during the school year. To accommodate this likely correlation, we estimated participation in each of these types of professional development simultaneously in an HLM framework. Specifically, the Level 1 models predicted hours of participation in professional development as a function of the kind of professional development taken (using a dummy variable for each type and a suppressed intercept). In the Level 2 models, we used teacher- and school-level characteristics to predict hours of participation in each type of professional development. Level 3 included measures of state policy attributes. This formulation allowed the residual terms or random effects (unmeasured factors associated with participation in professional development) for each of these three models to be correlated (i.e., to follow a multivariate normal distribution). Our Level 1 model was, in a sense, a measurement model describing the relationship between our latent and observed data (for a description of using hierarchical models for latent variables, see Raudenbush & Bryk, 2002).
While ideally we would have liked to include schools as a separate level of analysis, too few high school math and science teachers were sampled within each school to distinguish teacher and school characteristics separately. Therefore, the Level 2 portion of our models includes teacher characteristics as well as school variables. The Level 3 portion of each model includes measures of state policy attributes. Refer to Appendix B for a more detailed description of our models.
LIMITATIONS ON CAUSAL ANALYSIS
Although our conceptual model reflects hypotheses about causal relationships, our data were not based on experimental designs and did not follow the same sample of teachers and schools over time. Thus, we are not able to draw causal conclusions about the relationships between policy attributes and teacher participation in professional development (see Mosteller & Boruch, 2002; Whitehurst, 2002). However, the data we used was appropriate for estimating associations and generating strong hypotheses about cause and effect (see Berends & Garet, 2002; Cook, 1999; Cook, Habib, Phillips, Settersten, Shagle, & Degirmencioglu, 1999; Foorman, Francis, Fletcher, & Schatschneider, 1998).
After controlling for teacher and school characteristics, we found small, but significant, relationships between school and state policies and teacher participation in professional development. Furthermore, these relationships were strongest for content-focused professional development and weaker for other forms of professional development, as we hypothesized. We also found that, while state policies are important in promoting content-focused professional development for teachers, more localized policies (at the school level) demonstrate stronger relationships with teacher participation in professional development. As expected, these results differed for mathematics and science, with stronger relationships observed between policy attributes and teacher participation in content-focused professional development in mathematics, a high-stakes subject area. By contrast, relationships between policy attributes and teacher participation in content-focused professional development in science, a low-stakes subject area, were smaller in terms of effect size, less likely to reach statistical significance, or worked in the opposite direction as predicted by theory.
Policy Attributes & Professional Development Participation for Math Teachers
As Table 1 demonstrates, in mathematics, a high-stakes subject area, several measures of the school policy environment were important in predicting teacher participation in content-focused professional development. Math teachers who reported that they had more influence over school policya measure of authorityparticipated in 1.8 more hours of content-focused professional development than teachers who reported that they had less influence over school policy (p < .001). In terms of effect size, this coefficient represents a 6% of a standard deviation increase in participation in content-focused professional development.4 Teachers in schools where principals observe and supervise teachers more oftena measure of powerwere likely to participate in .7 additional hours of content-focused professional development (p < .05), a 2% of a standard deviation increase in participation. Stability was also an important predictor of participation in content-focused professional development. Teachers in schools where the principal had been at the school for at least three yearsa measure of stabilitywere likely to take about two more hours of content-focused professional development than teachers in schools with newer principals (p < .01). This is equivalent to a 4% standard deviation increase in participation in content-focused professional development. At the school level, our measure of consistency was not significant.
Some school policies also positively predicted teacher participation in teaching strategies professional development, but not professional development focused on classroom management. We interpret these findings to suggest that the school policies that are likely to foster teacher participation in learning opportunities focused on content or pedagogy are not the same factors that might motivate teachers to take professional development focused on classroom management. Our findings specifically demonstrate that math teachers who report more influence over school policya measure of authoritywere likely to take an additional hour of teaching strategies professional development (p < .001). This represents a 3.5% of a standard deviation increase in participation in teaching strategies professional development. Also similar to the findings for content-focused professional development participation, teachers in schools with experienced principalsprincipals who had been at the school for three or more years (a measure of stability)were likely to take about 2.5 more hours of teacher strategies professional development than teachers at a school with a less experienced principal (p < .001). This is equivalent to a 5% of a standard deviation increase in professional development focused on teaching strategies. However, as the percentage of teachers who had been at a school for three or more years increased, the number of hours of participation in teaching strategies professional development decreased slightly (by about .02 hours; p < .05). This is equivalent to a 2% of a standard deviation decrease in participation in teaching strategies professional development. A similar, negative relationship exists between the stability of teachers at a school and participation in classroom management professional development. We would expect this, as more experienced teachers tend to have better mastery of classroom management (Evertson & Harris, 1999; Freiberg, Stein, & Huang, 1995).
This study demonstrates that school policies are, to some degree, related to teachers seeking content-focused professional development, though we note that the size of the effects we find here are small. However, given the research-based evidence of content-focused professional developments link with improved instruction and student achievement, our results suggest potential for policies to manipulate this important teacher behavior. Similarly, our findings that school policies work to move teachers into professional development that involves the learning of teaching strategies is also encouraging. While the research base is not as strong in terms of linking particular pedagogy (separate from content) to student achievement, there is a strong intuition that such a link may exist, and certainly the idea of developing more engaging and diverse teaching strategies has the potential to serve teachers and students well.
At the state level, only one policy attribute significantly corresponded to increases in participation in content-focused professional development. Our consistency measure, an indicator of whether the state uses criterion-referenced assessments that are aligned to state standards in mathematics at the high school level, was significantly related to a 1.7 hour increase in teacher participation in content-focused professional development (p < .05). To interpret the size of this effect, teachers in states that use criterion-referenced assessments that are aligned to state standards in high school mathematics participate in 3% of a standard deviation more hours of content-focused professional development. No other state policy attributes were significantly predictive of teacher participation in any of the three types of professional development measured in this study.5
Policy Attributes & Professional Development Participation for Science Teachers
As we hypothesized, fewer policy attributes (at both the school and state levels) were significantly related to teacher participation in professional development for science teachers. In science, currently a low-stakes subject area, only two of the school-level policy attributes were significantly related to science teachers participation in content-focused professional development, and one worked in the opposite direction we hypothesized (see Table 2). When teachers worked in a school where the principal had been in his or her role for at least three years (a measure of stability), they were likely to take another 1.7 hours of content-focused professional development (p < .05). This is equivalent to a 3% of a standard deviation increase in content-focused professional development, a slightly smaller effect when compared to the results associated with the same measure of stability for math teachers participation in content-focused professional development. Also, when additional barriers existed for the dismissal of teachers (a measure of power, or lack of it), science teachers were more likely to participate in content-focused professional development (by an additional .5 hours; p < .001). The size of this effect can be interpreted as a 4% of a standard deviation increase in content-focused professional development participation. This finding suggests that this measure of power works in the opposite direction than predicted by the policy attributes theory (i.e., the weaker the power in the policy environment as measured by this variable, the more likely science teachers are to take content-focused professional development).
Science teachers participation in teaching strategies professional development was also correlated with one of our measures of stability. When teachers teach in a school with a more experienced principalsomeone who has served as principal in the school for at least three yearsthey are also more likely to participate in teaching strategies professional development (an increase of 1.2 hours, p < .05), which yields a 2% of a standard deviation increase in teaching strategies professional development. None of the school-level policy attributes predicted participation in professional development focused on classroom management.
At the state level, our measure of consistencyan indicator of whether or not a state uses criterion-referenced assessments that are aligned to state standardswas associated with increased participation in content-focused professional development in math. However, we did not find this relationship for science. In fact, none of our measures of the state policy environment were significantly related to science teachers participation in any of the types of professional development used in our analyses.
DISCUSSION & CONCLUSIONS
In our study of a national sample of high school math and science teachers, we find that state- and school-level policy environments are associated with teachers taking the type of professional development that is most associated with improved teaching and learningprofessional development focused on subject matter content. In general, we find that attributes of state and school policy environments are more predictive of teachers participation in effective professional development when teachers teach in a high-stakes subject area. We also find that policies promoting consistency in the form of alignment between standards and assessments are perhaps the most important types of policies that states can adopt to encourage teachers to participate in effective professional development. Nevertheless, our analyses suggest that this finding may only be relevant to high-stakes subject areas (such as mathematics). We also find that state policies are often diffused, and that school policy environments tend to demonstrate the strongest relationship between policies and teacher behaviors.
Specific to this study, state and school policy environments matter most for mathematics teachers and their participation in professional development focused specifically on math. At the school level, measures of authority, power, and stability were predictive of math teacher participation in content-focused professional development, though our analyses yielded relatively small effects. Measures of power and stability were predictive of high school science teachers participation in professional development focused specifically on science content; however, our power measure worked in the opposite direction than suggested by the policy attributes theory. Our results suggest that barriers to dismissing teachersa measure that indicates a lack of powerwas positively related to content-focused professional development. Therefore, we conclude that school policy environments are more influential in high-stakes subject areas.
At the state level, this trend continues. The only state policy measure related to teacher participation in content-focused professional development was our measure of consistency, which identified states that use criterion-referenced assessments aligned to state standards. However, we found this relationship only in mathematics, a high-stakes subject area. Nevertheless, science education has recently come under the microscope as NCLB required states to adopt science content standards for science and to begin testing students yearly in science. Although it is up to individual states to decide if students performance on these assessments will be a factor in determining whether a district or school is meeting adequate yearly progress, the renewed focus on the implementation of science standards and aligned tests is likely to change the way state policy influences science teachers participation in professional development.
In relation to our question about how state policy environments influence teacher participation in professional development, we find that consistencya policy attribute that was not significantly related to teacher participation in professional development at the school levelis an important policy attribute at the state level. This finding helps explain one of the ways that alignment between standards and assessments, a fundamental component of standards-based reform (Smith & ODay, 1991), may work to improve the educational system. As reforms such as NCLB are mobilized through state educational policies, perhaps the most important function of these state policies is to create alignment and consistency. States can accomplish this by establishing coherent policies and programs that facilitate alignment between federal education legislation, expected outcomes for students, and state standards. These policies should demonstrate how each educational strategy supports and reinforces the others (Clune, 1998; Doolittle, Herlihy, & Snipes, 2002; Porter & Brophy, 1988; Smith & ODay, 1991). In the current environment of NCLB, policies that bear the hallmark of consistency should represent the alignment of state standards with assessments, curricula, and teacher professional development (Cohen, 1995; Corcoran, 1997; Schmidt & Prawat, 1999).
This study also sought to determine the importance of state policies in relation to school policies. Others have documented that standards-based reform is initiated at the state level and interpreted and reshaped at the local level (Dutro, Fisk, Koch, Roop, & Wixson, 2002). Our results support this idea and suggest that the local interpretation and implementation of policy is most directly related to teacher behaviors. We know from previous work that state policy is important for establishing targets and providing supports, but a major component of the current standards-based reform movement is to allow districts and schools to choose their own strategies for improving teacher and student learning. While our results do not find overwhelming independent relationships with state policy, they do show that school-level policies are associated with teachers having the type of learning experiences the literature suggests are related to better teaching and therefore increased student achievement.
Our findings suggest that the alignment of at least two major components of standards-based reform (standards and assessments) appear to be associated with greater teacher participation in content-focused professional development, which is another major component of standards based reform. Nevertheless, researchers have argued that aligned professional development (and to some extent, aligned assessment) is the major piece of standards-based reform that has not been delivered (Resnick, 2006). Under the current system of top-down accountability and standards-based reform, it is likely that states are best equipped to adopt policies that create consistency and alignment with federal educational mandates.
Of course, the path from state policy to teachers behaviors is long and non-linear. While our results demonstrate an important link in the relationship between teacher behavior and the policy attributes theory, we acknowledge that not all of the policy attributes we measured yielded significant results. Furthermore, and the statistically significant effects associated with teacher participation in professional development and policy attributes were generally small in size. As such, we view our work here as a first step in examining the link between teacher behaviors and the policy attributes theory on a national level, and we recommend that future research further examine the path from state and school policies to teacher behaviors. Notwithstanding the weaknesses in our study, our finding that consistency at the state level is significantly related to teacher participation in content-focused professional development contributes to a growing body of research that documents the importance of coherence and alignment in a teachers policy environment (e.g., Desimone, Smith, & Phillips, 2007). For example, in-depth studies of whole-school change efforts have documented that, when teachers are pulled in different directions by contrasting reforms, they often do not become committed to any reform and adhere to their usual practice. However, when the policy messages that teachers receive consistently push them in the same direction, teacher motivation to pursue learning experiences and implement the reforms called for by their school or district increases (e.g., Berends, Bodilly, & Kirby, 2002; Datnow, 2000; Desimone, 2002a).
While we interpret our findings in relation to specific NCLB policies, we remind readers that the components of NCLB we address here represent major tenets of all standards-based reformsthe alignment of standards and assessments. We argue that even as changes are made to NCLB by national education administrations, the main components of NCLB that embrace standards-based reform initiatives will likely remain intact. This is supported by the research demonstrating that the degree to which standards-based reform encourages coherence among policy initiatives is important in most policy environments, including pre-NCLB policies (Clune, 1998). Therefore, any standards-based reformwhether it is called NCLB or something differentis more effective when coherence among policy initiatives is achieved, as our results suggest.
Because NCLB and related reforms are making new demands on teachers, we see professional development as a critical mechanism by which the U.S. educational system can be improved and by which teachers accumulate the necessary tools to successfully navigate these new reforms and new educational mandates. Therefore, it is important that we find the most effective ways to encourage teachers to participate in the types of professional development most likely to improve their practice, and, in turn, student achievement. We suggest that education policies can provide important avenues that foster change and improvements in the current teaching force. Nevertheless, we find that these policies are only effective when they are related to positive outcomes for teacherssuch as increasing participation in high quality professional development. We offer our findings as a contribution to the understanding of how best to shape policy to provide the most useful opportunities for teacher learning, with the ultimate goal of increased student learning.
1. Even though English language arts was another high-stakes subject area with teachers who might have been included in our analyses, we focused only on mathematics teachers (mathematics being a high-stakes subject area) and science teachers (with science considered a low-stakes subject area). Based on prior research, math and science teachers are more likely to participate in content-focused professional development and less likely to participate in other types of professional development when they teach advanced math or science classes (Desimone, Smith, & Phillips, 2007). In this study, we were able to determine which types of courses a teacher most often teaches (advanced or otherwise) for two reasons: first, because math and science courses are often ordered hierarchically and sequentially, and second, math and science courses tend to be similar in content and in title between schools, districts, and states. For example, calculus tends to be similar in both name and content in every district and state, as does physics. However, other subjects, such as English language arts, are often not ordered hierarchically or sequentially, thereby limiting our ability to identify teachers who teach advanced courses in non-sequential subject areas. Furthermore, English language arts courses are often unique to school, district, or state contexts. For example, literature might be a basic course in one state and an advanced course in another. In an effort to control for the difficulty level of the courses any given teacher is most likely to teach, we limit our analyses to math and science teachers.
2. All composite measures were created using exploratory factor analysis in the form of principal components analysis with varimax rotation, which is appropriate given the data we use in this study (Schonemann, 1990; Steiger, 1990; Velicer & Jackson, 1990).
3. Due to the fact that several of our variablesespecially our policy attributes measures at the school and state levelsmeasure seemingly overlapping concepts, we tested for the presence of multicollinearity among the variables we used in our models. We found that none of our measures (including all of the policy attributes measures at both the school and state levels) were highly correlated; therefore, we concluded that our measures did not overlap in significant ways and could be used to test distinctly different concepts in our analyses.
4. Effect sizes are determined by multiplying the coefficient by its standard deviation and then dividing the product by the standard deviation of the dependent variable.
5. In addition to examining the relationships between state- and school-level policy attributes and teacher participation in various types of professional development, we also tested all possible interactions in preliminary models. These tests included interactions among the state-level policy attributes, interactions among school-level policy attributes, cross-level interactions between state- and school-level policy attributes, as well as cross-level interactions between both state- and school-level policy attributes and teacher characteristics. None of these interactions yielded significant results for either our sample of math teachers or science teachers. Therefore, in an effort to present parsimonious models, we have excluded these interactions from the full models presented in Tables 1 and 2.
American Federation of Teachers (2001). Making standards matter 2001: A fifty-state
report on efforts to implement a standards-based system. Washington, DC: Author.
Ball, D. L. (1990). The mathematical understandings that prospective teachers bring to teacher
education. Elementary School Journal, 90(4), 449466.
Berends, M. (2000). Teacher-reported effects of New American Schools design:
Exploring relationships to teacher background and school context. Education Evaluation and Policy Analysis, 22, 6582.
Berends, M., Bodilly, S., & Kirby, S. (2002). Facing the challenges of whole-school
reform: New American Schools after a decade. Santa Monica, CA: RAND.
Berends, M., Chun, J., Schuyler, G., Stockly, S., & Briggs, R. J. (2002). Challenges of
conflicting school reforms: Effects of New American Schools in a high-poverty district. Santa Monica, CA: RAND.
Berends, M., & Garet, M. (2002). In (re)search of evidence-based school practices: Possibilities for integrating nationally representative surveys and randomized field trials to inform educational policy. Peabody Journal of Education, 77(4), 28-59.
Berman, P., & McLaughlin, M. W. (1975). Federal programs supporting educational
change. Vol. IV: The findings in review. Santa Monica, CA: RAND. (ED 108 330).
Bruk, A. S., Sebring, P. B., Kerbow, D., Rollow, S., & Easton, J. Q. (1998). Charting Chicago school reform. Boulder, CO: Westview Press.
Bybee, R. (1993). Reforming science education: Social perspectives and personal reflections.
New York: Teachers College Press.
Carnoy, M., & Loeb, S. (2003). Does external accountability affect student outcomes? A cross-state analysis. Education Evaluation and Policy Analysis, 24(4), 305-332.
Center on Education Policy (2007). Choices, changes, and challenges: Curriculum and instruction in the NCLB Era. Washington DC: Center on Education Policy.
Chatterji, M. (2002). Models and methods for examining standards-based reforms and accountability initiative: Have the tools of inquiry answered pressing questions on improving schools? Review of Educational Research, 72(3), 345-386
Clune, W. (1998). Toward a theory of systemic reform: The case of nine NSF statewide systemic initiatives (National Institute for Science Education Research Monograph No. 16). Madison, WI: National Institute for Science Education.
Cohen, D. K. (1990). A revolution in one classroom: The case of Mrs. Oublier. Educational Evaluation and Policy Analysis, 12(3), 311329.
Cohen, D. K. (1995, December). What is the system in systemic reform? Educational Researcher, 24(9), 11-17, 31 (EJ 519 251).
Cohen, D. K. (1996). Standards-based school reform: Policy, practice, and performance. In H. Lass (Ed.), Holding schools accountable: Performance-based reform in education (pp. 99-127). Washington, DC: Brookings Institution.
Cohen, D. K., & Hill, H. C. (2000). Instructional policy and classroom performance: The mathematics reform in California. Teachers College Record, 102(2), 294-343.
Cook, T. D. (1999). Considering the major arguments against random assignment: An analysis of the intellectual culture surrounding evaluation in American schools of education. Paper presented at the Harvard faculty seminar on experiments in education. Evanston, IL: Northwestern University.
Cook, T. D., Habib, F., Phillips, M., Settersten, R. A., Shagle, S. C., & Degirmencioglu, S. M. (1999). Comers school development program in Prince Georges County, Maryland: A theory-based evaluation. American Educational Research Journal, 36(3), 543-597.
Cook, T. D., Murphy, R., & Hunt, D. (2000). Comers school development program in Chicago: A theory-based evaluation. American Educational Research Journal, 37(2), 535-597.
Corcoran, T. B. (1997, February). Evaluating systemic reform. Prepared for Teacher Enhancement Electronic Community Hall. Retrieved from http://teech. Tec.edu/papers/papers/Corcoran.htm
Corcoran, T. B., Shields, P., & Zucker, A. (1998, March). The SSIs and professional development for teachers: Evaluation of NSFs Statewide Systemic Initiatives (SSI) Program (Monograph). Arlington, VA: National Science Foundation, Division of Research, Evaluation, and Communication, Directorate for Education and Human Resources.
Datnow, A. (2000). Power and politics in the adoption of whole school reform models.
Educational Evaluation and Policy Analysis, 22, 357374.
Datnow, A., & Stringfield, A. (2000). Working together for reliable school reform.
Journal of Education for Students Placed at Risk, 5, 183204.
Desimone, L. (2002a). How can comprehensive school reform models be successfully
implemented? Review of Educational Research, 72, 433479.
Desimone, L. (2002b). What makes comprehensive school reform successful. Review of Educational Research, 72(3), 433-479.
Desimone, L. (2006). Consider the source: Response differences among teachers, principals
and districts on survey questions about their education policy environment. Educational Policy, 20(4), 640-676.
Desimone, L., Birman, B., Porter, A., Garet, M., & Suk Yoon, K. (2003). Improving teachers in-service professional development in mathematics and science: The role of postsecondary institutions. Educational Policy, 17(5), pp. 613-649.
Desimone, L., Porter, A. C., Birman, B. F., Garet, M S., & Suk Yoon, K. (2002, October). How do district management and implementation strategies relate to the quality of the professional development that districts provide to teachers? Teachers College Record 104(7), 1265-1312.
Desimone, L., Smith, T., Frisvold, D. (2007). Has NCLB improved teacher and teaching quality for disadvantaged students? In A. Gamoran (Ed.), Standards-based reform and the poverty gap: Lessons for "No Child Left Behind" (pp. 89-119). Washington, D.C: Brookings Institution.
Desimone, L., Smith, T., Hayes, S., & Frisvold, D. (2005). Beyond accountability and average math scores: Relating multiple state education policy attributes to changes in student achievement in procedural knowledge, conceptual understanding and problem solving in mathematics. Educational Measurement: Issues and Practice, 24(4), 518.
Desimone, L., Smith, T., & Phillips, K. (2007). Does policy influence mathematics and science teachers participation in professional development? Teachers College Record, 109(5), 1086-1122.
Desimone, L., Smith, T., & Ueno, K. (2006). Are teachers who need sustained, content-focused professional development getting it? An administrators dilemma. Educational Administration Quarterly, 42(2), 179-215.
Doolittle, F., Herlihy, C., & Snipes, J. (2002). Foundations for success: Case studies of how urban school systems improve student achievement. New York: MDRC for the Council of Great City Schools.
Dutro, E., Fisk, M., Koch, R., Roop, L., & Wixson, K. (2002). When state policies meet local district contexts: Standards-based professional development as a means to individual agency and collective ownership. Teachers College Record, 104, 787811.
Education Week. (2000). Quality counts 2000: Building blocks for success. Education Week, 19(17). Retrieved from http://www.edweek.org/sreports/qc00/
Elmore, R. F. (1993). The role of local school districts in instructional improvement. In S. H. Fuhrman (Ed.), Designing coherent educational policy: Improving the system (pp. 96124). San Francisco: Jossey-Bass.
Elmore, R. F., Abelmann, C., Fuhrman, S. (1996). The new accountability in state education reform: From process to performance. In H. Ladd (Ed.), Holding schools accountable: Performance based reform in education (pp. 65-98). Washington, DC: Brookings Institution.
Elmore, R. F., Peterson, P. L., & McCarthey, S. J. (1996). Restructuring in the classroom: Teaching, learning, & school organization. San Francisco: Jossey-Bass.
Evertson, C. M. & Harris, A. H. (1999). Support for managing learning-centered classrooms: The classroom organization and management program. In H. J. Freiberg (Ed.), Beyond behaviorism: Changing the classroom management paradigm (pp. 59-74). Boston: Allyn and Bacon.
Finn, C., Jr., & Petrilli, M. (2000). The state of state standards. Washington, DC: The Thomas B. Fordham Foundation.
Foorman, B., Francis, D., Fletcher, J., & Schatschneider, C. (1998). The role of instruction in learning to read: Preventing reading failure in at-risk children. Journal of Educational Psychology, 90, 37-55.
Freiberg, H. J., Stein, T. A., & Haung, S. (1995). The effects of classroom management intervention on student achievement in inner-city elementary school. Educational Research and Evaluation, 1, 33-66.
Fullan, M. (2001). Leading in a culture of change. San Francisco: Jossey-Bass.
Garet, M., Birman, B., Porter, A., Yoon, K., & Desimone, L. (2001). What makes professional development effective? Analysis of a national sample of teachers. American Education Research Journal, 38, 915945.
Grant, S. G., Peterson, P. L., & Shojgreen-Downer, A. (1996). Learning to teach mathematics in the context of systemic reform. American Educational Research Journal, 33(2), 502541.
Haggstrom, G. W., Darling-Hammond, L., & Grissmer, D. (1988). Assessing teacher
supply and demand. Santa Monica, CA: Rand.
Hannaway, J., & Kimball, K. (1997). Reports on reform from the field: District and state survey results. Washington, DC: U.S. Department of Education.
Hanushek, E. & Raymond, M. (2004). The effective school accountability systems on the level and distribution of student achievement. Journal of European Economic Association, 2(2), 406-415.
Herman, J., & Golan, S. (1991). Effects of standardized testing on teachers and learning: Another look. Los Angeles: University of CaliforniaLos Angeles, National Center for Research on Evaluation, Standards, and Student Testing.
Huberman, A. M., & Miles, M. B. (1984). Rethinking the quest for school improvement: Some findings from the DESSI study. Teachers College Record, 86, 3454.
Kennedy, M. M. (1998). Form and substance in in-service teacher education (Research Monograph No. 13). Arlington, VA: National Science Foundation.
Loucks-Horsley, S., Hewson, P. W., Love, N., & Stiles, K. E. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press.
Louis, K. S., & Marks, H. M. (1998). Does professional community affect the classroom? Teachers work and student experiences in restructuring schools. American Journal of Education, 106, 532575.
Louis, K., Marks, H., & Kruse, S. (1996). Teachers professional community in restructuring schools. American Educational Research Journal, 33, 757798.
Ma, L. (1999). Knowing and teaching elementary mathematics: Teachers understanding of fundamental mathematics in China and the United States. Mahwah, NJ: Lawrence Erlbaum.
Massell, D. (1998). State strategies for building capacity in education: Progress and continuing challenges. CPRE Research Report Series, RR-41. Philadelphia: Consortium for Policy Research in Education, University of Pennsylvania.
Massell, D. (2001). The theory and practice of using data to build capacity: State and local strategies and their effects. In S. Furhrman (Ed.), From the capitol to the classroom: Standards-based reform in the states. One hundredth yearbook of the National Society for the Study of Education, Part II. Chicago, IL: University of Chicago Press.
McLaughlin, M. W., & Talbert, J. E. (2001). Professional communities and the work of high school teachers. Chicago: University of Chicago Press.
Mirel, J. (1994, Fall). School reform unplugged: The Bensenville New American School project, 1991- 93. American Educational Research Journal, 31(3), 481- 518.
Mosteller, F. & Boruch, R.. Eds. (2002). Evidence matters: Randomized trials in education research (pp. 150-178). Washington, DC: Brookings.
Murphy, J., & Hallinger, P. (1992). The principalship in an era of transformation. Journal of Educational Administration, 30(3), 77-78 (EJ 449 894).
National Commission on Teaching and Americas Future. (1996, September). What matters most: Teaching for Americas future. New York: Author.
National Council of Teachers of Mathematics. (1991). Professional standards for the teaching of school mathematics. Washington, DC: Author.
National Research Council. (1996). National science education standards. Washington, DC: National Academy Press.
Newmann, F., Smith, B., Allensworth, E., & Bryk, A. (2001, January). School instructional program coherence: Benefits and challenges. Chicago: Consortium on Chicago School Research.
Porter, A. (1994). National standards and school improvement in the 1990s: Issues and promise. American Journal of Education, 102, 421-449.
Porter, A. C. (2002, October). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3-14.
Porter, A. C., & Brophy, J. E. (1988). Good teaching: Insights from the work of the Institute for Research on Teaching. Educational Leadership, 45(8), 75-84.
Porter, A. C., & Smithson, J. (2001). Are content standards being implemented in the classroom? A methodology and some tentative answers. In S. Fuhrman (Ed.), From the capitol to the classroom: Standards-based reform in the states; One hundredth yearbook of the National Society for the Study of Education, Part II (pp. 60-80). Chicago: University of Chicago Press.
Porter, A. C., Archbald, D., & Tyree, Jr., A. (1990). Reforming the curriculum: Will empowerment policies replace control? In S. H. Fuhrman & B. Malen (Eds.), The politics of curriculum and testing: The 1990 yearbook of the Politics of Education Association (pp. 1136). New York: Falmer.
Porter, A. C., Floden, R., Freeman, D., Schmidt, W., & Schwille, J. (1988). Content determinants in elementary school mathematics. In D. Grouws & T. Cooney (Eds.), Perspectives on research on effective mathematics teaching (pp. 96113). Reston, VA: National Council of Teachers of Mathematics.
Porter, A. C., Kirst, M. W., Osthoff, E. J., Smithson, J. L., & Schneider, S. A. (1993, October). Reform up close: An analysis of high school mathematics and science classrooms. Madison, WI: University of WisconsinMadison.
Purkey, S. C., & Smith, M. S. (1983). Effective schools: A review. The Elementary School Journal, 83(4), 427-452.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
Rech, J., Hartzell, J., & Stephens, L. (1993). Comparisons of mathematical competencies and attitudes of elementary education majors with established norms of a general college population. School Science and Mathematics, 93(3), 141145.
Resnick, L. B. (2006). Making accountability really count. Educational Measurement: Issues and Practice, 25(1), 33-37.
Romberg, T., Zarinnia, E., & Williams, S. (1989). The influence of mandated testing on mathematics instruction: Grade 8 teachers perceptions. Madison, WI: University of WisconsinMadison, National Center for Research in Mathematical Science Education.
Ross, S. M., Alberg, M., & Nunnery, J. (1999). Selection and evaluation of locally developed versus externally developed schoolwide programs. In G. Orfield & E. H. DeBray (Eds.), Hard work for good schools: Facts not fads in Title I reform (pp. 14758). Cambridge, MA: Harvard University, The Civil Rights Project.
Ross, S. M., Henry, D., Phillipsen, L., Evans, K., Smith, L., & Buggey, T. (1997, March). Matching restructuring programs to schools: Selection, negotiation, and preparation. School Effectiveness and School Improvement, 8, 4571.
Schmidt, W. H., & Prawat, R. S. (1999, Spring). What does the Third International Mathematics and Science Study tell us about where to draw the line in the top-down versus bottom-up debate? Educational Evaluation and Policy Analysis, 21(1), 85-91.
Schonemann, P. H. (1990). Facts, fictions, and common-sense about factors and components. Multivariate Behavioral Research, 25(1), 53-59.
Schwille, J., Porter, A., Alford, L., Floden, R., Freeman, D., Irwin, S., & Schmidt, W. (1988). State policy and the control of curriculum decisions. Educational Policy, 2(1), 29-50.
Sebring, P. B., & Bruk, A. S. (2000, February). School leadership and the bottom line in Chicago. Phi Delta Kappan, 81(6), 440-443.
Sizer, T. R. (1992). Horaces school: Redesigning the American high school. Boston: Houghton Mifflin.
Smith, M. S., & ODay, J. (1991). Systemic school reform. In S. H. Fuhrman & B. Malen (Eds.), The politics of curriculum and testing: The 1990 yearbook of the Politics of Education Association (pp. 233-267). Bristol, PA: Falmer.
Smith, T. M. (2006). How do state-level induction and standers-based reform policies affect induction experiences and turnover among new teachers? American Journal of Education, 113, 273-309.
Smith, T., & Desimone, L. (2003). Do changes in patterns of participation in teachers professional development reflect the goals of standards-based reform? Educational Horizons, 81, 119129.
Spillane, J. (1996). School district matter: Local educational authorities and state instructional policy, Educational Policy, 10(1), 63-87.
Spillane, J., & Jennings, N. (1997). Aligned instructional policy and ambitious pedagogy: Exploring instructional reform from the classroom perspective. Teachers College Record, 98, 450481.
Spillane, J., & Louis, K. (2002). School improvement processes and practices. In J. Murphy (Ed.), The educational leadership challenge: Redefining leadership for the 21st century (pp. 83-104). Chicago: University of Chicago Press.
Steiger, J. H. (1990). Some additional thoughts on components, factors, and factor indeterminacy. Multivariate Behavioral Research, 25(1), 41-45.
Sykes, G. (1996). Reform of and as professional development. Phi Delta Kappan, 77, 465489.
Timar, T. (1989, December). The politics of school restructuring. Phi Delta Kappan, 71( 4), 165-
Tirosh, D., & Graeber, A. O. (1989). Preservice elementary teachers explicit beliefs about multiplication and division. Educational Studies in Mathematics, 20, 7996.
Tyack, D., & Tobin, W. (1994). The grammar of schooling: Why has it been so hard to change? American Educational Research Journal, 31, 453479.
United States Department of Education. National Center for Education Statistics [NCES] (1999). The NAEP 1996 Technical Report. NCES 1999-452. N. L. Allen, J. E. Carlson, & C. A. Zelenak. Washington, DC: Author.
United States Department of Education. National Center for Education Statistics [NCES] (2002). Schools and staffing survey: 1999-2000. Overview of the data for public, private, public charter, and Bureau of Indian Affairs elementary and secondary schools. NCES 2002-213. K. J. Gruber, S. D. Wiley, S. P. Broughman, B. A. Strizek, & M. Burian-Fitzgerald. Washington, DC: Author.
Velicer, W. F. & Jackson, D. N. (1990). Component analysis versus common factor-analysis: Some further observations. Multivariate Behavioral Research, 25(1), 89-95.
Webb, N. L., & Romberg, T. A. (1994). Reforming mathematics education in Americas cities: The urban mathematics collaborative project. New York: Teachers College Press.
Weber, M. (1947). The theory of social and economic organization. New York: Oxford University Press.
Wenglinsky, H. (2000). How teaching matters: Bringing the classroom back into discussions of teacher quality (Policy Information Center Report). Princeton, NJ: Educational Testing Service.
Wenglinsky, H. (2002). How schools matter: The link between teacher classroom practices and student academic performance. Education Policy Analysis Archives, 10(12), 1-31.
Whitehurst, G. (2002, February 28). Statement before subcommittee on education reform, Committee on Education and the Workforce, U.S. House of Representatives, Washington, DC.