Does Policy Influence Mathematics and Science Teachers’ Participation in Professional Development?
by Laura M. Desimone, Thomas M. Smith & Kristie J. R. Phillips - 2007
Recent research has shown the importance of professional development for teacher learning and has documented the qualities that make professional development effective for improved instruction and student achievement. But there is little research to suggest how the policy environment shapes teachers’ choices to participate in either “effective” or “ineffective” professional development. Because No Child Left Behind (NCLB) and related reforms are making new demands on teachers, and professional development is one of the critical mechanisms by which we intend to improve our educational system, it is important that we find the most effective ways to encourage teachers to participate in the types of professional development most likely to improve their practice—and, in turn, student achievement.
In describing the policy environment on several dimensions, we seek to discover which types of policies are more or less influential in moving teachers into the types of professional development that research has shown to be most effective for improved teaching and learning. In addition, we examine whether these relationships differ for a high-stakes subject, mathematics, and a low-stakes subject, science.
We characterize the policy environment based on a theory that suggests that certain attributes of the policy environment increase policy implementation: (1) authority—the extent to which a policy is persuasive; (2) power (or accountability)—rewards and sanctions attached to a policy; (3) consistency—how aligned a policy is with other elements in the policy system; and (4) stability—how stable actors and ideas in the policy environment are.
Our analyses answer two main questions. Do attributes of the policy environment—authority, power, consistency, and stability—influence the likelihood that teachers will participate in professional development with research-based features of effectiveness, rather than classroom management or no professional development? Is this relationship between policy attributes and professional development participation different for high-stakes subjects (e.g., mathematics) than for lower stakes subjects (e.g., science)?
Using a national sample of high school mathematics and science teachers from the Schools and Staffing Survey (SASS), we conduct a secondary analysis using a three-level hierarchical linear model (HLM) to predict teachers’ level of participation in different types of professional development activities.
We find that authority, not power, is associated with teachers taking the kind of professional development that we know improves teaching and learning—activities focused on subject matter content and instructional strategies, as well as active interactions with other teachers around curriculum and instruction. Similarly, we find that stability (measured by reduced teacher turnover), not the consistency of professional development with other reforms, is associated with taking effective professional development. We offer our findings to contribute to understanding how best to shape policy to provide the most useful opportunities for teacher learning.
This study focuses on the intersection of education policy environment and teachers participation in professional development. Recent research has shown the importance of professional development for teacher learning and has documented the qualities that make professional development effective for improved instruction and student achievement. But how do teachers come to participate in either effective or ineffective professional development? There is little research to suggest how the policy environment shapes teachers choices about which professional development activities to select.
In this analysis, we consider the emerging prominence of accountability and other policy mechanisms and ask to what extent particular policies are associated with teachers participation in effective professional development in a high-stakes subject, mathematics, and in a still low-stakes (though probably not for long) subject, science. In describing the policy environment on several dimensions, we seek to discover which types of policies are more or less influential in moving teachers into the types of professional development that research has shown to be most effective for improved teaching and learning.
Studies in the past several years have provided empirical evidence of what conventional wisdom has long espousedthat the most effective professional development activities for increasing teachers knowledge and skills and improving their teaching practice are those that (1) focus on subject matter content and how students learn that content, (2) are ongoing and sustained throughout the year, (3) are consistent with other activities, and (4) provide teachers with opportunities to actively interact and engage with each other around curriculum and instruction (e.g., Desimone, Porter, Garet, Suk Yoon, & Birman, 2002; Garet, Birman, Porter, Yoon, & Desimone, 2001; McLaughlin & Talbert, 2001). Further, research has indicated that participation in such activities is positively related to student achievement (Cohen & Hill, 2001; Kennedy, 1998). As a result, recent standards-based reforms, including NCLB, have focused on the importance of improving teaching quality through increasing the participation of teachers in effective or high-quality professional development that has these features of quality. This is in contrast to the much-maligned but ever-resilient and still prevalent one-shot workshop (Garet et al.), which is often focused on management, discipline, or administrative issues rather than on subject matter content.
If improvements in teaching and learning rely to a large extent on teachers experiences in professional development (Corcoran, Shields, & Zucker, 1998; Loucks-Horsley, Hewson, Love, & Stiles, 1998; National Commission on Teaching & Americas Future [NCTAF], 1996; Sykes, 1996), then it is critical to understand which policies work best to foster teachers participation in the most effective forms of professional development.
Do attributes of the policy environmentauthority, power, consistency, and stabilityinfluence the likelihood that teachers will participate in professional development focused on subject matter content, instructional strategies, and interactive learning, rather than on classroom management or no professional development at all? Is this relationship between policy attributes and professional development participation different for high-stakes subjects (e.g., mathematics) than for lower-stakes subjects (e.g., science)? Our analyses address these questions.
A FRAMEWORK FOR STUDYING THE ACCOUNTABLITY ENVIRONMENT: THE POLICY ATTRIBUTES THEORY
We characterize the policy environment based on a policy attributes theory that has been used to study systemic reform (Clune, 1998), comprehensive school reform (Berends, Chun, Schuyler, Stockly, & Briggs, 2002; Desimone, 2002), and teacher decision making in general (Porter, Archbald, & Tyree, 1990; Porter, Floden, Freeman, Schmidt, & Schwille, 1988). The policy attributes theory provides an organizing framework for describing the education policy environment. In this study, we focus on four policy attributes in particular: authority, power, consistency, and stability.1
The first attribute, authority, is the extent to which a policy is accepted and persuasive to those who have to implement itusually principals and teachers. Authority can be realized when a policy becomes part of teacher or school norms, or has the backing of a well-respected institution (e.g., the American Federation of Teachers) or individual (e.g., a well-liked principal or district superintendent). Authority can also be achieved through the participation of implementers in the design of the policy; for example, when teachers play a role in the shaping of the policies they are to implement, the policy may become persuasive to teachers through their own involvement and buy-in to the policy (Spillane & Jennings, 1997). Such buy-in has long been noted to be a key component of teachers adoption and implementation of reform efforts (e.g., Datnow, 2000; Tyack & Tobin, 1994).
The second attribute is power. Power is achieved through rewards and sanctions. NCLB power mechanisms include the evaluation of schools based on student achievement scores and the evaluation of teachers based on their credentials. If teachers follow a policy because of its power, they implement it only because of the threat of rewards or sanctions. If, however, they follow a policy because of its authority, they implement it because they have been persuaded that it is a good idea. Both authority and power have been shown to be related to policy implementation (e.g., Berends, Bodilly, & Kirby, 2002; Berman & McLaughlin, 1975; Datnow, 2000; Louis & Marks, 1998). There is evidence to suggest that power sometimes results in shallow, short-term implementation, whereas authority is more likely to result in longer term and deeper implementation (Desimone, 2002).
Consistency, the third attribute, is the extent to which a policy is aligned with other policies in the same school, district, and state, and with the perceptions and beliefs of its implementers (see Spillane, 2004, for a discussion of the role of teacher perceptions in policy implementation). One of the most common criticisms from teachers is that a new reform they are asked to follow contradicts a previous reform (or worse yet, a concurrent reform; Datnow & Stringfield, 2000; Ross, Alberg, & Nunnery, 1999). An example of this is the inconsistency between some comprehensive school reform programs, such as the Coalition of Essential Schools (CES), with high-stakes testing. CES requires the development of a philosophy of learning, and its teaching and learning goals take several years to fully develop. Thus, standards-based reforms yearly requirements for standardized testing are in tension with the proper cycle of CES, which does not expect to see gains on test scores for several years (Muncey & McQuillan, 1996; Smith et al., 1997). This creates a conundrum for teachers and principals. Alternatively, when reforms are aligned with each other and are pushing teachers in the same direction (i.e., when they are consistent), they are more easily and readily implemented (Newmann, Smith, Allensworth, & Bryk, 2001).
The fourth policy attribute that we consider is the stability of the policy environment. How long policies and people remain a stable part of the policy landscape has a significant influence on the level and quality of implementation (Huberman & Miles, 1984). Savvy teachers know that reforms come and go. As a result, they often greet a new policy with a wait and see attitude, expecting that it will disappear over the horizon like so many of its predecessors (Ross et al., 1997). The longer a policy remains in place, or the longer a principal, district superintendent, or teachers remain in their jobs, the more stable the policy environment. This type of stability provides a supportive context for policy implementation (Berends, Chun, et al., 2002).
GREAT EXPECTATIONS: PREDICTIONS ABOUT HOW POLICY ATTRIBUTES WILL INFLUENCE PROFESSIONAL DEVELOPMENT PARTICIPATION
Data from a U.S. Department of Education study of a national probability sample of teachers indicated that most teachers select their own professional development, and most experience fewer than 8 hours a year in content-focused professional development (Garet et al., 2001). However, an analysis of the U.S. Department of Educations Schools and Staffing Survey showed that from 19931994 to 19992000, teachers increased the number of hours of content-focused professional development that they took (Smith & Desimone, 2003). What caused this increase? How influential is policy in shaping teachers decisions about which professional development to take?
We expect policy to be influential, based on three assumptions. First, professional development is one of the major mechanisms of school improvement. Second, there is a research consensus on the qualities of the most effective professional development. Third, current policies are largely designed to improve teaching. Therefore, we would expect that one way that policies work is to increase teachers participation in effective professional development. That is, we expect policy mechanisms to be moving teachers away from professional development focused on non-content- and instruction-related topics such as classroom management and discipline, and toward activities that have features of effective professional development, such as a focus on content and strategies to teach content, and opportunities for active/interactive learning (e.g., Cohen & Hill, 2001; Garet et al., 2001; Kennedy, 1998). Trend studies have shown, for example, that teachers in 19992000 took more sustained, content-focused professional development in mathematics than their counterparts in 19931994 (Smith & Desimone, 2003). Did policy attributes play a role in this change?
Specifically, we hypothesize that in high-stakes subjects such as mathematics, the attributes of policies have a stronger relationship with teachers taking content-focused professional development than in lower stakes subjects such as science, a subject that is not yet included in the calculation of whether a school is failing. If the policy attributes work as indicated by the theorymore of each improves implementationwe expect a high-stakes accountability environment to interact with the already-effective attributes for an even stronger effect. For example, if a teacher feels pressure to increase her students mathematics test scores, this pressure interacts with existing policies designed to encourage teachers to participate in professional development; thus, policies offering rewards and sanctions attached to test scores might play more of a role in moving the teacher into content-focused professional development, as compared with a subject area in which there is less external pressure, such as science. We would also expect that attributes of policies (e.g., power and authority) would be more intense in high-stakes subjects. In subjects such as science, in which external pressures are not as great, we would not expect the policy attributes to be as strong or as effective.
Further, of the four attributes, we expect authority to be the most salient. Previous research provides support for the idea that nothing is as powerful a motivator for implementing a policy than believing it is worthwhile and having a personal stake in it (Cohen, McLaughlin, & Talbert, 1993).
Despite the current emphasis on rewards and sanctions, there are few, if any, empirical studies of the comparative effects of power mechanisms and authority mechanisms such as participatory decision making and buy-in. We intend our study to (1) provide a comparative examination of the relationship between different policy attributes and their associations with teachers participation in professional development activities, and (2) explore whether the pressure of teaching a high-stakes subject makes these associations, if they exist, any stronger.
This study uses data from the Schools and Staffing Survey (SASS), a nationally representative sample of teachers and schools. The SASS is a random sample of schools stratified by state, public/private sector, and school level (Haggstrom, Darling-Hammond, & Grissmer, 1988).
Our analysis uses the restricted-use version of the 19992000 SASS public school survey; we use both the administrator questionnaires and the questionnaires administered to a linked random sample of teachers within each school. The total 19992000 SASS sample comprises about 52,000 elementary and secondary teachers. However, our analyses focus on public high school teachers whose main assignment fields are math and science, and their principals. Because one of our dependent variables is a composite measure indicating the number of hours that teachers participate in content-focused professional development, we restricted our analysis to specific content areas of teaching, and analyses for math and science teachers are conducted separately. Therefore, we report two different sets of resultsone for mathematics and one for science. The sample of high school teachers who reported mathematics as their main assignment field was 2,008; 1,819 high school teachers reported science as their main assignment field. Sample weights were used to compensate for the oversampling and undersampling of schools and teachers in the complex stratified survey design. Each teacher and administrator is weighted by the inverse of the probability of their selection to obtain unbiased estimates of the national population of public schools and teachers in the year of the survey.
In our analysis, we measure the relationship between teachers participation in professional development and attributes of the policy environment. We control for teacher and school characteristics that are likely to be related to the policy and professional development variables. Below we describe our measures. Table 1 provides the exact SASS questions that correspond to each of the teacher, principal, and school variables; shows how we code the variables; and provides the means and standard deviations (where applicable) for each of the two sampleshigh school math teachers and high school science teachers.
Dependent Variables: Participation in Professional Development
Professional development takes many different forms, with a wide range of quality and effectiveness. We focus on what the research has shown to be among the most salient characteristics for effects on teaching and learninga focus on subject matter content and how to teach that content, and professional development that allows teachers the opportunity to participate together and interact around curriculum and instruction (Cohen & Hill, 2001; Garet et al., 2001; Kennedy, 1998; Louis, Marks, & Kruse, 1996). We contrast this with non-content- or instruction-related professional development activities that cover topics concerning student discipline and classroom management.
We acknowledge that a better measure of quality of a particular professional development activity would include data on all its characteristics. Ideally, we would like to know the full range of characteristics that define a teachers experiences in professional development, including span of time, contact hours, how aligned it was with other activities, whether teachers participated with other teachers from their school or grade, and so on. Such data are not yet available on a national level for multiple teacher activities, but we have confidence in the measures of quality we chose because focus on content, instruction, and active/interactive learning are consistently found to be leading features of quality (e.g., Cohen & Hill, 2001; Garet et al., 2001; Kennedy, 1998), and classroom management/discipline is equally consistently shown to not be related to improved teaching and student achievement (e.g., Loucks-Horsley et al., 1998).2
In our analyses, we predict participation in four different types of professional development: (1) content-focused professional development (measured as number of hours that teachers participated in professional development related to in-depth study and standards in the teachers main assignment field, either mathematics or science); (2) professional development focused on teaching strategies (computed from the number of hours teachers participated in professional development dealing with methods of teaching, using technology for instruction, and methods of student assessment); (3) professional development focused on classroom management (measured with one item indicating the number of hours teachers spend in professional development dealing with discipline and classroom management); and (4) interactive professional development (a composite measure indicating whether teachers made observational visits to other schools; individually or collaboratively researched a topic of interest to them professionally; participated in regularly scheduled collaboration with other teachers on issues of instruction, mentoring, and/or peer observation and coaching; and participated in a network of teachers). The first three measures of participation in professional development were created by summing across the midpoint of five duration categories for each of these three types of professional development (0 = did not participate; 4 = 8 hours or less; 12.5 = 916 hours; 24.5 = 1732 hours; and 40 = 33 hours or more). The final outcome variable, participation in interactive professional development, was computed by summing across the five items listed above. For each item, respondents indicated whether they had participated in any of the collaborative activities in the past 12 months (0 = no, 1 = yes).
The 19992000 SASS teacher and administrator questionnaires contain items related to all four policy attributes. Of these items, we were able to construct three measures of authority, three measures of power, one measure of consistency, and two measures of stability.
Authority. Our first authority measure has to do with teacher reports of how much influence they think teachers in their school have over school-level policies. We created a composite indicator of teacher influence over school policy by summing teachers responses (1 = no influence and 5 = a great deal of influence) to the following seven items: setting performance standards, establishing curriculum, determining the content of in-service professional development programs, evaluating teachers, hiring new full-time teachers, setting discipline policy, and deciding how the school budget will be spent. Factor analysis confirmed that all these items are associated with a single construct, and the composite indicator created by summing the items has a Cronbachs alpha reliability of .80. We grand-mean centered the construct; that is, the construct measures the difference between an individual teachers perception of the level of influence that teachers have over policy in their school, and the mean of teachers sampled in the SASS who are in our mathematics or science analysis.
Our second measure of authority reflects teacher reports of their level of control over classroom practices. We created a composite variable from the six items that ask teachers how much control they have over planning the following in their classroom: selecting textbooks and other instructional materials; selecting content, topics, and skills to be taught; selecting teaching techniques; evaluating and grading students; disciplining students; and determining the amount of homework to be assigned (1 = no control and 5 = complete control). Factor analysis confirmed that each of these items loads on a single construct (which is an indication that the separate items are all measuring the same dimension or construct), and the composite indicator was created by summing the items (Cronbachs alpha = .77). As with the authority measure of control over school policy, we grand-mean centered the teacher influence over classroom policy indicator.
The third and final measure of authority describes principal reports of the degree to which teachers take leadership roles in designing and/or implementing professional development activities. The composite measure, teacher leadership in professional development, was created by summing across two items: administrator reports indicating (1) how often professional development is planned by teachers, and (2) how often it is presented by teachers in the school or district (0 = never, 1 = rarely, 2 = sometimes, 3 = frequently, and 4 = always). Cronbachs alpha for this composite was .70.
Power. In this study, our power variables are intended to capture sanctions that are associated with a schools policy environment. Each of our three measures of power come from principals reports of policies and practices at their school, and as with the measures of authority, each of the measures is grand-mean centered. Our first measure, evaluation of professional development, is a composite sum of two items. Principals were asked to report (1) how often professional development for teachers is evaluated for evidence of improvement in teacher classroom practices, and (2) how often it is evaluated for evidence of effects on student achievement. Possible responses were 0 = never, 1 = rarely, 2 = sometimes, 3 = frequently, and 4 = always (Cronbachs alpha = .87).
The second measure of power is a composite, barriers to teacher dismissal. This composite is a sum of responses to a question that asks about the extent to which each of the following six items can be considered barriers to dismissing poor or incompetent teachers: (1) personnel policies, (2) termination decisions not upheld by third party adjudicators, (3) inadequate teacher assessment documentation, (4) tenure, (5) teacher associations and organizations, and (6) that dismissal is too stressful and uncomfortable for those involved. Principals reported that each of these items either was (coded 1 = yes) or was not (coded 0 = no) a barrier to dismissal in their school (Cronbachs alpha=.69). The more barriers to firing teachers, the less powerful the policy environment is; we would expect teachers to feel more safe and secure in their jobs, and less at risk for negative sanctions (such as being fired), if there were many barriers to the principal firing them.
Our final measure of power is principals reports of how often they supervised and evaluated their faculty and other staff. Response categories were 1 = never, 2 = once or twice a month, 3 = once or twice a week, and 4 = every day.
Consistency. We measure the consistency of the policy environment with principal reports of how well the content of professional development for teachers is aligned with school-, district-, and state-level policies. We created a composite from nine items derived from two separate questions in SASS. Principals were asked, How important is each of the following in determining the in-service professional development activities of teachers in this school? (1) special state-level initiatives, (2) district-level initiatives or district improvement plan, (3) school improvement plan, (4) implementation of state or local ACADEMIC standards, (5) implementation of state or local SKILLS standards, and (6) teacher preference. Answers were reported in a range from 1 to 5, where 1 = not important at all and 5 = very important. These six items were combined with three additional items associated with the following question: How often is professional development for teachers at this school (1) designed or chosen to support the schools improvement goals, (2) designed or chosen to support the district's improvement goals, and (3) designed or chosen to support the implementation of state or local standards? (0 = never, 1 = rarely, 2 = sometimes, 3 = frequently, and 4 = always). Factor analysis confirmed that all nine of these items load on a single construct, and the composite indicator was created by summing the items (Cronbachs alpha reliability = .80).
According to the policy attributes theory, stability represents the extent to which people, circumstances, and policies remain constant over time. The SASS does not necessarily allow us to examine the stability of circumstances or policies, but we can, to some extent, examine the stability of the schools labor force over time. Our two measures of stability involve the frequency of principal and teacher turnover at schools. Our first measure is a dummy variable coded 1 if a principal has been at a school for 3 or more years and coded 0 if the principal has been there for less than 3 years. Our second measure is a percentage of teachers who participated in the SASS who have been at their current school for at least 3 years. Our stability of teachers variable is grand-mean centered in the analyses.
Teacher Background Characteristics
We include control variables in our models for several characteristics of teachers that previous research has shown to be related to teaching behaviors, including their professional development participation. We would expect full-time teachers to invest more heavily in their teaching than part-time teachers; similarly, we expect teachers in midcareer and early career to be more active in seeking learning opportunities and welcoming new reforms than teachers in the later stages of their careers (Berends, 2000). Further, we would expect teachers with advanced subject matter degrees to be more likely than their colleagues without such degrees to feel comfortable seeking out more content-related professional development, and we would expect that teachers teaching an advanced class would be more likely to take professional development focused on content and/or instruction to meet the advanced needs of their students. In addition, teachers without full certification would likely be taking professional development to fulfill their certification requirements.
Thus, in our analysis, we control for the following: whether a teacher is a regular full-time teacher (as opposed to a part-time teacher); a teachers total years of experience (grand-mean centered); years of experience squared (grand-mean centered); teachers education level; whether the teacher teaches an advanced class in mathematics or science; and teachers certification level (full, partial, or no certification). Teachers level of education is measured not only in terms of the highest degree achieved but also in terms of the content focus of their degree. Specifically, our categories for math teachers are as follows: bachelors degree or beyond in math (reference category); bachelors degree or beyond in math education; minor in math or bachelors degree or beyond in science (a related subject); and no major or minor in math or science. Similarly, our categories for science teachers are bachelors degree or beyond in science (reference category); bachelors degree or beyond in science education; minor in science or bachelors degree or beyond in math (a related subject); and no major or minor in science or math.
Teaching advanced classes was measured differently for math and science. If teachers taught at least one class in advanced algebra, analytic geometry, precalculus, or calculus, they were considered to be teaching at least one advanced class in math (as opposed to those teachers who did not teach any of these classes). However, because it is less clear which science classes should be considered advanced from the categories listed in the SASS, we created three dummy variables to better understand the relationship between teaching advanced science classes and science teachers participation in professional development. The three categories include whether teachers teach at least one class in physics, chemistry, and biology. Teachers of other types of science classes are used as the reference category. We expected the physics and chemistry teachers to represent advanced science classes, but given that biology is offered at many different levels (U.S. Department of Education, 1999), we did not necessarily consider it a proxy for advanced science class, but rather treated it as a way of distinguishing different types of science teachers.
We expected more professional development participation in high-poverty districts than in low-poverty districts because of the proliferation of programs and federal government funding (Elmore, 1993). Similarly, we expected more participation in urban than suburban or rural districts because of the higher concentration of teachers and more specialized programs (Hannaway & Kimball, 1997). Thus, we include a control for whether the school is urban, rural, or suburban. We also control for the level of poverty of the student population, measured by the percentage of students who are eligible for free and reduced lunch (grand-mean centered). Missing values for school poverty were imputed using STATA statistical software, based on the values of 17 items dealing with school climate.3 An imputation flag was created and included in all statistical models.
Our analyses are presented in two stages for both the high school mathematics teacher and the high school science teacher samples. For both stages of these analyses, we use a three-level hierarchical linear model (HLM) to predict teachers level of participation in different types of professional development activities. Type of professional development is Level 1, teacher is Level 2, and state is Level 3. The models for math and science use the same variables, with the exception of two teacher background characteristics: teacher education level and whether a teacher teaches advanced classes. Both of these variables have been constructed to account for subject-specific nuances in teacher education and patterns of student course teaching; therefore, changes to these respective variables have been made to more accurately accommodate math- and science-specific models.
For the first stage of the analysis, we predict hours of participation in professional development as a function of the type of professional development participated in: (1) content-focused professional development, (2) professional development focused on teaching strategies, and (3) professional development focused on classroom management. Although we could have modeled each of these as a separate dependent variable, we would have been making the implicit assumption that the taking of one kind of professional development was unrelated (i.e., uncorrelated) to the taking of other kinds of professional development. It is more likely, however, that decisions regarding participation in one type of professional development are conditional on how much professional development of other types has either been taken or is anticipated to be taken during the school year. To accommodate this likely correlation, we estimate participation in each of these types of professional development simultaneously in an HLM framework.
Specifically, the Level 1 model predicts hours of participation in professional development as a function of the kind of professional development taken (using a dummy variable for each type and a suppressed intercept). Then in the Level 2 models, we use teacher- and school-level characteristics to predict hours of participation in each type of professional development. This formulation allows the residual terms or random effects (unmeasured factors associated with participation in professional development) for each of these three models to be correlated (i.e., to follow a multivariate normal distribution). Our Level 1 model is, in a sense, a measurement model describing the relationship between our latent and observed data (hours of participation vs. the ordinal categories within which teachers responded; see Raudenbush & Bryk, 2002, for a description of using hierarchical models for latent variables.).
Because requirements for professional developmentas well as the incentives for taking different kinds of professional developmentvary across states, we included a state-level random effect for each kind of professional development in Level 3 of our analysis. Although ideally, we would have liked to have included school as a separate level of analysis, there were too few high school math and science teachers sampled within each school to distinguish teacher and school characteristics separately.
The analysis of the relationship between policy attributes and interactive professional development was conducted separately so that we could interpret the outcomes (hours of participation vs. number of interactive participations of different types) with respect to these more intuitive metrics. Please see the appendix for a more detailed description of the models.
As Table 1 shows, on average, professional development participation, teacher and school characteristics, and the policy environment for math and science teachers is remarkably similar. The only notable difference is that there are more science teachers with a major in science (58%) than there are math teachers with a major in math (34%). We expected that science teachers would have more control over classroom practices because math is a high-stakes subject, and there is likely more pressure to teach to the states content standards. There is little difference, however, between the mean score for classroom control for math teachers (4.09) and the mean score for classroom control for science teachers (4.18). As expected, both math and science teachers report having less influence over school policy than over their own classroom practices. Both math and science teachers spend, on average, a bit more time in professional development focused on teaching strategies (22.42 hours for math and 24.49 hours for science teachers) than content (20.65 hours for math and 20.07 hours for science teachers); teachers spent the least amount of time in professional development focused on classroom management (5.32 hours for math and 5.64 hours for science teachers).
We move to the predictive analysis to answer our main research questions: Do attributes of the policy environmentauthority, power, consistency, and stabilityincrease the likelihood that teachers will participate in professional development focused on subject matter content, instructional strategies, and interactive learning? Is this relationship between policy attributes and professional development participation different for math than for science? Table 2 shows the results of estimating the relationships between policy attributes and math teachers participation in the four types of professional development. Table 3 shows the results of the same estimation for science teachers.
Teachers who report having more of an influence on school policy are more likely to engage in interactive professional development (b = .271, p < .000 for math teachers and b = 313, p < .000 for science teachers). For math teachers, influence on school policy is also associated with an increase in taking professional development focused on math (b = .555, p < .001). For science teachers, more school policy influence is associated with higher participation rates in professional development that covers teaching strategies (b = .223, p < .044). These coefficients have modest relationships with professional development. For example, an increase in reported influence from 3 (some influence) to 4 (moderate influence) results in a 12.5-hour (per year) increase in content-focused professional development for math teachers, and about a 5.5-hour increase (per year) in professional development on teaching strategies for science teachers.4
For math teachers, involvement in planning and presenting professional development is positively associated with taking math-related professional development (b = .546, p < .003)for example, a 12.5-hour increase for a movement from rarely to sometimesbut the data do not show a similar relationship for science teachers. Planning and presenting professional development is associated with an increase in participation in activities focused on teaching strategies for both math teachers (b = .442, p < .001) and science teachers (b= .451, p < .001). We find no relationship between teachers influence on classroom practices and their participation in any type of professional development. This is the case for both math and science teachers.
Principal reports of the extent to which professional development for teachers is evaluated for evidence of improvement and for evidence of student achievement are associated with a decrease in science teachers taking professional development in science (b = -.285, p < .051; see Table 2) and marginally associated with a decrease in math teachers taking professional development in math (b = 0.239, p < .078; see Table 1).
Increased barriers to firing teachers indicates a less powerful accountability environment. This variable is associated with (1) math teachers taking a bit less classroom management-focused professional development (b = -.125, p < .022)1 hour less for every additional barrier (see Table 1)and (2) science teachers taking more interactive professional development (b = .043, p < .013), though this relationship is very small. An increase in principal reports of supervising and evaluating faculty is not significantly related to teachers participation in any of the four types of professional development.
Principals reported on how consistent the professional development activities of teachers were with initiatives, standards, and teacher preferences at the state, district, and local level. This variable was found to be unrelated to either math or science teachers participation in any of the four types of professional development.
Principal turnover in a school is unrelated to teachers participation in professional development. However, teacher turnover does have a significant association with participation. Specifically, an increase in the percent of teachers in a school who had been there for 3 or more years is associated with an increase in content-focused professional development for both math (b = .019, p < .005; see Table 1) and science teachers (b = .022, p < .003). This means, for example, that a 25% increase in stability translates into a 10- and 12.5-hour increase in content-focused professional development for math and science teachers, respectively. Similarly, increased teacher stability is associated with an increase in participation in professional development addressing teaching strategies for both math (b = .013, p < .004) and science (b = .016, p < .001) teachers. In addition, the more stability there is among the teaching staff, the less classroom management-focused professional development math teachers take (b = -.014, p < .020).
CONSIDERATIONS FOR INTERPRETATION
In our analysis, we operationalize the policy attributes framework (see Berends, Chun, et al., 2002; Clune, 1998; Desimone, 2002; Porter et al., 1990, 1988) to measure the relationship between authority, power, consistency, and stability on teachers propensity to take the type of professional development advocated by current reform efforts: specifically, professional development that is focused on subject matter content and how to teach that content, and that allows teachers the opportunity to interact and engage with each other around curriculum and instruction.
Instead of measuring state- and district-level policies, we measure principal and teacher self-reports of how state, district, and school policies are operationalized at the school and classroom level. We acknowledge the importance of the state and district role in setting and shaping policy. Standards-based reform is initiated at the state level, then interpreted and reshaped at the district level (Dutro, Fisk, Koch, Roop, & Wixson, 2002).5 Our analysis, however, is limited to the more proximal principals and teachers. Although a more comprehensive study of policy effects on teachers would consider multiple levels in the system, one strength of this study is that instead of focusing on distal policies, we focus on proximal school- and classroom-level policies. Theories about loose-coupling (e.g., Meyer & Rowan, 1978; Weick, 1976) and the importance of school and teacher interpretation of state and district policies (Spillane, 2004) would suggest that proximal interpretations are the most important for predicting effects.
Further, although we could imagine more comprehensive and holistic measures of the policy attributes, the measures we use are reasonable proxies. Our confidence in them is further increased by the fact that the relationships with control variables are in the expected directions. For example, as shown in Table 2, teachers in urban schools participate more in content-focused professional development. This reflects previous research indicating that school size and government programs afford increased professional development opportunities for teachers in urban schools (Hannaway & Kimball, 1997); similarly, teachers in rural schools are less likely to participate in interactive professional development, which makes sense because there are far fewer teachers, schools, and networks.
The survey measures, although necessarily limited and narrow in scope, are the type of survey items shown to have the most validity and reliabilitycomposite measures that ask behaviorally based questions rather than perceptions or opinions (Mayer, 1999). Further, the tradeoff with depth is that our sample is a national sample of math and science teachers, which increases confidence in our ability to generalize our findings.
Assumptions in our interpretations of results assume that among teacher choices for professional development are opportunities to focus on content, instruction, interaction, and classroom management and discipline (among other options), and that teachers choose among these options. National data on district and school professional development offerings support this assumption (e.g., Garet et al., 2001), though individual experiences of teachers may differ.
DO ATTRIIBUTES OF THE POLICY ENVIRONMENT INCREASE THE LIKLIHOOS THAT TEACHERS WILL TAKE EFFECTIVE PROFESSIONAL DEVELOPMENT?
We find that authority and stability play more of a role than power and consistency in moving teachers into taking professional development that has at least one of the characteristics of high-quality professional development, as defined in the literature. Specifically, professional development that focuses on content or teacher strategies, and/or that involves interactive learning is more highly associated with authority and stability than with power and consistency.
Persuasion, Not Power
Is co-optation through the threat of sanctions more effective than building cooperation through buy-in and joint decision making? If authority is the proverbial carrot, where participation and buy-in work as incentives for action, and power is the proverbial stick, where the threat of sanctions or the promise of rewards are the incentives for action, our analysis shows that the carrot is more effective than the stick. Both math and science teachers who are more involved in setting school policyincluding establishing the content of in-service professional development and evaluating teachers, curriculum, and standardsare more likely to participate in professional development that includes observing other schools and classrooms, participating in teacher networks, and taking part in regularly scheduled collaborations and mentoring. This supports other research that has shown that when teachers take an active role in designing their policy environment, collaborative and interactive learning opportunities are often at the core (Cohen et al., 1993; Little, 1982; Rosenholtz, 1991).
Another measure of authorityteachers planning and presenting during professional developmentwas associated with an increase in both math and science teachers taking professional development focused on instructional strategies. Both influence on school policy and planning and presenting were associated with math teachers increased participation in content-focused professional development. Thus, we find support for the idea that teacher involvement through persuasionactive participation in planning, building buy-in through participatory managementis associated with more participation in effective types of professional development. This is consistent with previous research, which suggests that teacher involvement in planning contributes to the provision of and teachers participation in high-quality professional development (Elmore, 1993; Floden et al., 1988; Loucks-Horsley et al., 1998).
Does power have the same effect? In our study, it does not. In fact, we find in one case that power has the opposite effect from what policy designers probably want it to have. When principals report more evaluation of teachers based on their teaching and their students achievement, both math and science teachers report taking less content-focused professional development. This implies that the threat of evaluation may serve as a disincentive for teachers to seek out challenging learning opportunities.
Other measures of power had null or inconsistent findings. A different measure of power might show varying results. Specifically, it would be instructive to measure power from the teachers perspective rather than that of the principal (as we do in the current study), to see if teachers perception of power is associated with their behavior. Still, our findings that authority is associated with teacher behavior but power is notor is, rather, a negative forceare consistent with other work showing that policy implementation that operates through power is usually weaker than implementation that operates through authority: the active participation and buy-in of teachers (Desimone, 2002). Further, these findings are consistent with recent work that has shown that formal rewards and sanctions instituted as part of the policy system do not serve as powerful motivators for teachers as do the intrinsic rewards of student success (Desimone & LeFloch, 2004).
Stability Trumps Consistency
Is constancy in terms of the stability of principals and teachers more important than the consistency of the professional development activities with other reforms in the school and district? We expected that the consistency of professional development with teacher preferences and with state, district, and local initiatives and standards would be associated with increased participation. In our data, it is not. This contradicts previous research that suggests the importance of the coherence and consistency of reforms with each other (Newman et al., 2001). In interpreting these results, one must wonder whether principal and teacher experiences of consistency are equivalent. Although principals and teachers often respond similarly in indicating problems in their schools and districts (Sweeney & Pinckney, 1983), some research indicates that there may be a divergence in their perceptions about positive policy environment characteristics (Bingham & White, 1993; Desimone, 2006). It could be that a teacher-reported measure of consistency would be more associated with teachers professional development.
Unlike consistency, teacher stability proved to be significantly associated with teachers participation in both content-focused and instruction-related professional development for math and science teachers. It makes sense that teachers are more likely to invest in their own learning if they are attached to a school and feel a sense of community, which is more likely when the same teachers spend more time at the same school (Louis & Marks, 1998).
Similar Policy Effects for Math and Science Teachers, Except for Authority
The only notable difference between math and science teachers was that authority plays more of a role for math teachers than for science teachers. Specifically, two measures of authorityteacher involvement in school policy and teachers planning and presenting during professional developmentwere related to math teachers taking more content-focused professional development, but not to science teachers. We expected to find more differences between math and science teachers. NCLB has created such a high-stakes environment for math that we anticipated that the effect of the policy attributes would be magnified. We find this to be true only of authority. Still, it implies that when teachers play an active role in shaping their schools policy and their own experiences in professional development, they are more likely to experience the type of professional development that is advocated by reformsand that we know is associated with improved teaching and learning. That we find this to be the case for math but not science teachers suggests that there may be a high-stakes environment policy attributes interaction that operates to motivate math teachers to seek the type of professional development most likely to help students meet state content standards requirements.
In our study of a national sample of math and science teachers, we find that authority, not power, is associated with teachers taking the kind of professional development that we know improves teaching and learningactivities focused on subject matter content and instructional strategies and that include active interactions with other teachers around curriculum and instruction. Similarly, we find that stability (measured by reduced teacher turnover), not the consistency of professional development with other reforms, is associated with taking effective professional development.
This analysis suggests that authority and stability may play more of a role than power or consistency in fostering teacher participation in professional development that is focused on content and that has opportunities for interaction. Further research should explore more comprehensive measures of the policy attributes and their possible interactions. Still, the findings are consistent with earlier work that notes the critical role that teacher participation, buy-in, and development of a stable community play in implementing any new reformand the potentially negative role that rewards and sanctions can play. Because NCLB and related reforms are making new demands on teachers, and professional development is one of the critical mechanisms by which we intend to improve our educational system, it is important that we find the most effective ways to encourage teachers to participate in the types of professional development most likely to improve their practice, and in turn, student achievement. We offer our findings to contribute to understanding how best to shape policy to provide the most useful opportunities for teacher learning.
LEVEL 1 MODEL
LEVEL 2 MODEL
LEVEL 3 MODEL
1 Specificity is also a component of the policy attributes framework. Specificity is the clarity and level of detail of a particular policy. Because we are not studying one policy in particular, but rather the policy environment as a whole, we do not include specificity in our analysis.
2 We do not intend to suggest that assistance with classroom management and discipline is not an appropriate and useful activity. But given the low percentage of teachers who participate in the type of professional development that research shows to be effective in improving teaching and learning, we contrast the more common classroom management professional development with content, instruction, and interactive professional development.
3 The 17 school climate items are student tardiness, student absenteeism, students cutting class, physical conflicts among students, robbery or theft, vandalism of school property, student pregnancy, student use of alcohol, student drug abuse, student possession of weapons, student disrespect for teachers, students dropping out, student apathy, lack of parent involvement, poverty, students come to school unprepared to learn, and poor student health.
4 To translate the beta coefficients into hours, we use the standard deviations reported in Table 1. To translate the beta coefficient (.555) for the relationship between influence on school policy and content-focused professional development for math teachers, .555*22.89 (the standard deviation for content-focused professional development for math teachers, reported in Table 1) = 12.5 hours. Similarly, b =.223 for the relationships between influence on school policy and professional development on teacher strategies for science teachers translates into .223*.24.47 (the standard deviation for professional development on teaching strategies for science teachers, reported in Table 1) = 5.45.
5 Using the SASS district data would make the number of respondents drop so low that we would be unable to estimate the relationships of interest.
Berends, M. (2000). Teacher-reported effects of New American Schools design: Exploring relationships to teacher background and school context. Education Evaluation and Policy Analysis, 22, 6582.
Berends, M., Bodilly, S., & Kirby, S. (2002). Facing the challenges of whole-school reform: New American Schools after a decade. Santa Monica, CA: RAND.
Berends, M., Chun, J., Schuyler, G., Stockly, S., & Briggs, R. J. (2002). Challenges of conflicting school reforms: Effects of New American Schools in a high-poverty district. Santa Monica, CA: RAND.
Berman, P., & McLaughlin, M. W. (1975). Federal programs supporting educational change. Vol. IV: The findings in review. Santa Monica, CA: RAND. (ED 108 330)
Bingham, R., & White, S. (1993). Explaining teacher/principal differences in evaluating schools. Journal of Educational Administration, 31, 7076.
Clune, W. (1998). Toward a theory of systemic reform: The case of nine NSF statewide systemic initiatives (National Institute for Science Education Research Monograph No. 16). Madison, WI: National Institute for Science Education.
Cohen, D. K., & Hill, H. C. (2001). Learning policy: When state education reform works. New Haven, CT: Yale University Press.
Cohen, D. K., McLaughlin, M. W., & Talbert, J. E. (Eds.). (1993). Teaching for understanding: Challenges for policy and practice. San Francisco: Jossey-Bass.
Corcoran, T., Shields, P., & Zucker, A. (1998, March). The SSIs and professional development for teachers: Evaluation of NSFs Statewide Systemic Initiatives (SSI) Program (Monograph). Arlington, VA: National Science Foundation, Division of Research, Evaluation, and Communication, Directorate for Education and Human Resources.
Datnow, A. (2000). Power and politics in the adoption of whole school reform models. Educational Evaluation and Policy Analysis, 22, 357374.
Datnow, A., & Stringfield, A. (2000). Working together for reliable school reform. Journal of Education for Students Placed at Risk, 5, 183204.
Desimone, L. (2002). How can comprehensive school reform models be successfully implemented? Review of Educational Research, 72, 433479.
Desimone, L. M. (2006). Consider the source: Response differences among teachers, principals, and districts on survey questions about their education policy environment. Educational Policy, 20, 640676.
Desimone, L., & LeFloch, K. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26, 122.
Desimone, L., Porter, A. C., Garet, M., Suk Yoon, K., & Birman, B. (2002). Does professional development change teachers instruction? Results from a three-year study. Educational Evaluation and Policy Analysis, 24, 81112.
Dutro, E., Fisk, M., Koch, R., Roop, L., & Wixson, K. (2002). When state policies meet local district contexts: Standards-based professional development as a means to individual agency and collective ownership. Teachers College Record, 104, 787811.
Elmore, R. F. (1993). The role of local school districts in instructional improvement. In S. H. Fuhrman (Ed.), Designing coherent educational policy: Improving the system (pp. 96124). San Francisco: Jossey-Bass.
Floden, R. E., Porter, A. C., Alford, L. E., Freeman, D. J., Irwin, S., Schmidt, W. H., et al. (1988). Instructional leadership at the district level: A closer look at autonomy and control. Educational Administration Quarterly, 24, 96124.
Garet, M., Birman, B., Porter, A., Yoon, K., & Desimone, L. (2001). What makes professional development effective? Analysis of a national sample of teachers. American Education Research Journal, 38, 915945.
Haggstrom, G. W., Darling-Hammond, L., & Grissmer, D. (1988). Assessing teacher supply and demand. Santa Monica, CA: Rand.
Hannaway, J., & Kimball, K. (1997). Reports on reform from the field: District and state survey results. Washington, DC: U.S. Department of Education.
Huberman, A. M., & Miles, M. B. (1984). Rethinking the quest for school improvement: Some findings from the DESSI study. Teachers College Record, 86, 3454.
Kennedy, M. M. (1998). Form and substance in in-service teacher education (Research Monograph No. 13). Arlington, VA: National Science Foundation.
Little, J. W. (1982). Norms of collegiality and experimentation: Workplace conditions of school success. American Educational Research Journal, 19, 325340.
Loucks-Horsley, S., Hewson, P. W., Love, N., & Stiles, K. E. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press.
Louis, K. S., & Marks, H. M. (1998). Does professional community affect the classroom? Teachers work and student experiences in restructuring schools. American Journal of Education, 106, 532575.
Louis, K., Marks, H., & Kruse, S. (1996). Teachers professional community in restructuring schools. American Educational Research Journal, 33, 757798.
Mayer, D. P. (1999). Measuring instructional practice: Can policymakers trust survey data? Educational Evaluation and Policy Analysis, 21, 2945.
McLaughlin, M. W., & Talbert, J. E. (2001). Professional communities and the work of high school teachers. Chicago: University of Chicago Press.
Meyer, J., & Rowan, B. (1978). The structure of educational organizations. In M. Meyer & Associates (Eds.), Environments and organizations (pp. 7198). San Francisco: Jossey-Bass.
Muncey, D. E., & McQuillan, P. J. (1996). Reform and resistance in schools and classrooms: An ethnographic view of the Coalition of Essential Schools. New Haven, CT: Yale University Press (ED 399 345).
National Commission on Teaching and America's Future. (1996, September). What matters most: Teaching for America's future. New York: Author.
Newmann, F., Smith, B., Allensworth, E., & Bryk, A. (2001, January). School instructional program coherence: Benefits and challenges. Chicago: Consortium on Chicago School Research.
Porter, A. C., Archbald, D., & Tyree, Jr., A. (1990). Reforming the curriculum: Will empowerment policies replace control? In S. H. Fuhrman & B. Malen (Eds.), The politics of curriculum and testing: The 1990 yearbook of the Politics of Education Association (pp. 1136). New York: Falmer.
Porter, A. C., Floden, R., Freeman, D., Schmidt, W., & Schwille, J. (1988). Content determinants in elementary school mathematics. In D. Grouws & T. Cooney (Eds.), Perspectives on research on effective mathematics teaching (pp. 96113). Reston, VA: National Council of Teachers of Mathematics.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
Rosenholtz, S. J. (1991). Teachers workplace: The social organization of schools. New York: Teachers College Press.
Ross, S. M., Alberg, M., & Nunnery, J. (1999). Selection and evaluation of locally developed versus externally developed schoolwide programs. In G. Orfield & E. H. DeBray (Eds.), Hard work for good schools: Facts not fads in Title I reform (pp. 14758). Cambridge, MA: Harvard University, The Civil Rights Project.
Ross, S. M., Henry, D., Phillipsen, L., Evans, K., Smith, L., & Buggey, T. (1997, March). Matching restructuring programs to schools: Selection, negotiation, and preparation. School Effectiveness and School Improvement, 8, 4571.
Smith, T., & Desimone, L. (2003). Do changes in patterns of participation in teachers professional development reflect the goals of standards-based reform? Educational Horizons, 81, 119129.
Smith, L. J., Maxwell, S., Lowther, D., Hacker, D., Bol, L., & Nunnery, J. (1997). Activities in schools and programs experiencing the most, and least, early implementation successes. School Effectiveness and School Improvement, 8, 125150.
Spillane, J. (2004). Standards deviations: How local schools misunderstand policy. Cambridge, MA: Harvard University Press.
Spillane, J., & Jennings, N. (1997). Aligned instructional policy and ambitious pedagogy: Exploring instructional reform from the classroom perspective. Teachers College Record, 98, 450481.
Sykes, G. (1996). Reform of and as professional development. Phi Delta Kappan, 77, 465489.
Sweeney, J., & Pinckney, R. (1983). Faculty management: The principals most important role. Spectrum, 1(3), 36.
Tyack, D., & Tobin, W. (1994). The grammar of schooling: Why has it been so hard to change? American Educational Research Journal, 31, 453479.
U.S. Department of Education, National Center for Education Statistics. (1999). Teacher quality: A report on the preparation and qualifications of public school teachers (NCES 1999-080). Washington, DC: U.S. Government Printing Office.
Weick, K. (1976). Educational organizations as loosely coupled systems. Administrative Science Quarterly, 21, 541552.