Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13
Topics
Discussion
Announcements
 

Do Incentive-Based Programs Improve Teacher Quality and Student Achievement? An Analysis of Implementation in 12 Urban Charter Schools


by Girija Kaimal & Will J. Jordan - 2016

Context: Policymakers have increasingly advocated for incentive-based approaches for improving urban schools.

Purpose of the study: Few studies have examined the implementation of incentive based approaches in the urban charter school context. This paper presents research findings from a 4-year longitudinal study of the implementation of a comprehensive incentive-based program for school improvement.

Setting: The study was conducted in cluster of 12 charter schools in a large urban school district.

Participants/Subjects: Participants in the study included classroom teachers, administrators, and staff in the charter schools; program staff from the school district; and consultants to the project.

Research Design: The study used a mixed methods design, which included a comparative case study, and quasi-experimental design.

Data Collection and Analysis: Qualitative data elements included semistructured interviews and observations of program activities. Quantitative data elements included the official classroom teaching evaluation scores, teacher attitudes survey data, standardized student achievement scores from state and district assessment programs, and researcher-developed surveys of teacher and administrator perceptions.

Results: The results suggest that, in addition to questions about the viability of this comprehensive incentive model, there were several challenges related specifically to the context of the schools that affected program implementation. These included: misconceptions about performance pay; difficulties learning, understanding and sustaining a complex professional development model; poor fidelity of program implementation; varying capacities of the leadership teams; inability to sustain the incentive component; differences in the contexts and missions of the participating schools; and high teacher turnover at the schools. All the 12 schools reported some improvement in student achievement which led to performance rewards; however, this was not substantial or consistent over the course of the 4 years of implementation. The program also did not impact teacher retention at the schools. Schools that benefited the most from the program demonstrated an alignment of goals between the program staff, school leadership, and teachers.

Conclusions/Recommendations: The study findings suggest that even comprehensive incentive-based models are not yet viable and have limited effectiveness. Educators needed significant salary increases to be incentivized by money and incentive-based models need to be simpler in design for participants to understand them. The aspect that was somewhat sustained was the professional development and collegiality among the teachers indicating that a professional community is what teachers appreciated most from the program.



INTRODUCTION


In the decades since A Nation Risk (Gardner, 1983), there has been widespread discontent about the weak quality of teaching and learning in American schools. Scholars have been especially concerned about the extent to which efforts to improve the conditions of education in urban schools have stagnated. While economists have generated evidence linking incentive-based systems with developing human capital, educational theorists often have not considered how this approach might apply to schools and teaching (Hanushek, Heckman, & Neal, 2002). In this paper, we review the literature on various forms of incentive-based reforms that have been introduced in schools; challenges in successful implementation; identify the most comprehensive model developed thus far; and; present findings from a multi-year study of this program. At present little is understood about the contextual factors that impact implementation and long-term outcomes of incentive based approaches. The gaps in the literature on incentives based reforms that this paper addresses include presenting findings from a longitudinal study of implementation (most studies present findings from about one year of implementation), identifying unique issues related to the context of charter schools, and changes in educator perceptions, understanding, and attitudes towards such models.


LITERATURE REVIEW


APPROACHES TO INCENTIVE-BASED REFORM


Numerous studies point to complex challenges of improving city schools (Bryk, Harding, & Greenberg, 2012; Fruchter, 2007; Lipman, 2004; Neild & Balfanz, 2006). There is also concern about a disincentive for the most talented people to become and remain teachers considering that teachers earn anywhere from 12%–25% less than their comparably educated and experienced professionals in other fields—even after adjusting average teacher incomes for fringe benefits (Mishel, Allegretto, & Corcoran, 2008). There is also research that has implied that a “teaching penalty” exists and that this is even more present in large urban school districts which are often underfunded, under resourced, and difficult to staff (Anyon, 2005; Cuban & Usdan, 2003; Kozol, 1991; Mirel, 1998). More recently reformers and policymakers have advocated for a market-based approach promoting incentive-based reforms to improve school quality, such as pay-for-performance (Gallagher, 2004). Both Race to the Top, an Obama Administration initiative, and No Child Left Behind from the Bush Administration have advocated for incentive-based school reforms. Some have found empirical linkages between incentive-based systems, instructional quality, and student achievement (Eberts, Hollenbeck, & Stone, 2002). Incentive-based programs currently take many forms throughout the nation. Most have the goal of rewarding teachers financially for the academic achievement of their students as measured by standardized test scores (Carr, 2008; Grier & Holcombe, 2008). Some are designed to attract effective teachers to hard-to-staff schools and subject areas. Other incentive programs pay for teachers’ input. Input-based incentives promote teachers’ growth and involvement in the school outside the classroom, engaging teachers in professional development and encouraging their acceptance of leadership roles in their schools and communities.


IDENTIFIED CHALLENGES IN SUCCESSFUL IMPLEMENTATION OF INCENTIVE MODELS


Both states and local districts have had difficulty implementing their pay-for-performance plans successfully. Challenges in implementation have related to designs of the programs; lack of buy-in from teachers and unions; limited understanding of incentive models; lack of funding for incentive payouts; poor rollout of programs; and lack of transparency in evaluation of teachers and criteria for incentives. For instance, Houston and Alaska have received criticism for the lack of transparency and clarity in their merit pay systems. Houston’s performance pay program was challenged for its measures of teacher competence, as many educators who were widely hailed as exemplary teachers received no payouts via the system. In addition, Houston’s failed accountability systems resulted in numerous embarrassments, such as teachers being paid mistakenly and then having to return the additional bonuses. The Alaska incentive system, similarly, was chided for schools having received performance pay even though they had not met adequate yearly progress, as determined by No Child Left Behind, for several consecutive years (Flannery & Jehlen, 2008).


No group contests the use of incentive-based systems more than the teacher unions; thus it is no wonder that school districts with strong teachers unions successfully resist merit-based reforms (Goldhaber, DeArmond, Player, & Choi, 2008). The National Education Association (NEA) has supported the input perspective advocating pay for teachers who have engaged in professional development activities, those who earned advanced degrees, and those who have taken on the responsibility of supporting new educators in a mentor capacity (Flannery & Jehlen, 2008). According to The American Federation of Teachers (AFT), only completion of the National Board for Professional Teaching Standards certificate was heralded as the standard of excellence through which teachers demonstrated their expertise and, therefore, justified additional pay (Feldman, 2000; Laverick, 2007). However, the AFT’s president has publicly supported New York City’s school-wide merit pay program because of its collaborative nature and fairness (Quaid, 2008).


When examining the idea of differentiating teaching compensation based on teaching quality, a lack of consensus on measures of teacher quality is a major issue. There is mounting evidence suggesting that teacher evaluations using multiple sources of data, which may include student testing, stand a better chance of producing substantial improvements in the validity of teacher evaluation systems. School leaders have become better at evaluating teachers using data from classroom observations and supervisory sessions (Andrejko, 2004; Danielson, 1996). There persists, nevertheless, a fear of opaque, subjective evaluation processes and validity problems inherent in formative data collection, combined with a lack of power sharing portrayed by some districts which has left opponents leery of including qualitative data as a merit pay determinant. Both national teachers unions also have rejected the idea of using student performance data to determine merit pay increases; the NEA supports the use of student performance data for the purpose of improving instruction rather than supporting decisions about teacher compensation (Flannery & Jehlen, 2008). Marshaling resources to fund incentive-based programs has also been a challenge. For example, Guilford County Schools in North Carolina initially paid for its “Mission Possible” program by increasing class size and not filling teacher assistant positions; later, they supplanted internal funding with the federal government’s Teacher Incentive Fund (Grier & Holcombe, 2008).


Districts which appear to be having success with their incentive-based programs owe part of that success to the collaborative efforts of administration and unions or, in right-to-work states, collaboration with teachers (Dillon, 2008; Grier & Holcombe, 2008). Such successful collaborations among stakeholders suggest that: (1) The way that incentive-based pay programs are described and implemented requires finesse, sophistication, public engagement, and input from key players; (2) power must be shared; and (3) even the best systems will need constant monitoring and tweaking (Carr, 2008; Dillon, 2008; Flannery & Jehlen, 2008; Lavy, 2007).


Although direct empirical evidence is lacking, some researchers believe incentive-based reforms might more likely be effective when integrated with strong school leadership, professional development, reliable analyses of student performance, and ongoing feedback (Odden, 2004; Odden & Kelley, 2002). Building collective capacity through consensus of teachers is preferable to leadership through mandate (Darling-Hammond, Bullmaster, & Cobb, 1995). Shared responsibility for instructional leadership can help staff to pursue collective organizational goals, thus leading to improved curriculum and instruction (Leithwood, Tomlinson, & Genge, 1996; Silins, Mulford, Zarins, & Bishop, 2000). DuFour (2004) advanced the notion of professional learning communities which could inspire action-oriented collaboration and collective inquiry in pursuit of constant improvement of a school. In professional learning communities, teachers see themselves as responsible for the learning of all students in the school, not only their classrooms (Schlechty, 2009).


OVERVIEW OF THE INCENTIVE-BASED MODEL THAT WAS RESEARCHED FOR THE PRESENT STUDY


It is in this context that this paper emerged from the evaluation of the participating urban school district’s implementation of this incentive-based model. The study was conducted as part of a federally funded evaluation of a comprehensive incentive-based model that had been implemented in several districts and schools throughout the United States. The incentive based model focused on linking teacher instructional development and financial incentives for teachers and school leaders. The goals of the incentive-based model were to improve student achievement by increasing teacher quality and principal effectiveness, calibrating teacher and principal compensation to their “productivity,” and increasing the number of high-quality leaders and teachers in hard-to-staff schools. This newly designed incentive-based model was more comprehensive than a traditional merit pay system and attempted to account for the identified deficiencies of previous incentive-based programs cited in the literature. Traditional merit pay systems tend to be narrowly focused on linking cash payouts to student achievement, with less attention paid to building support for teachers and school leaders. For example, an incentive-based model like the one examined in this study, required participation in professional development experiences, opting into multiple career pathways for teaching and differentiated leadership, mentorship for all staff, accountability, and ongoing instructional support. These activities were conducted as part of the professional development offerings and participation in these components of the program was expected and linked to the financial incentives. Moreover, instead of calculating average year-to-year gains in student achievements using standardized scores, this model assessed growth through value-added measures. The developers of the incentive-based model advocated for rethinking teacher compensation (traditional merit pay model) and hierarchical organization of school personnel. According to the incentive model developer, the theory of action for this model was that differentiated payments, above the regularly salary could “spark” school-wide improvement by motivating teachers and leaders toward excellence. During implementation, the developer’s role was to provide training to appropriate school district staff. “Train-the-trainer” was the phrase used to describe this approach, where national experts worked with cadres of educators comprised of representatives from each school implementing the new incentive-based model. School-based representatives were expected to learn the intricacies of the incentive-based model and, in turn, conduct training back at the school and transfer the appropriate knowledge and skills to their colleagues.


In addition, the incentive model developers conducted annual surveys of teacher perceptions (which were disseminated to the local sites, including these authors) to assess attitudes towards the incentive-based model. They also provided ongoing support to school districts on implementation, monitoring for fidelity of implementation, training materials for teacher evaluation, and calculations of the financial incentive component. The model involved incorporating school-wide professional development, along with teacher classroom observation protocols, value-added assessments, and an elaborate system of cash payouts. The payouts for teachers and principals in the incentive-based model followed the retention-as-priority approach (implying a focus on retaining teachers in the school) offering year-long instructional and professional support and expecting ongoing accountability.


The design and content of the teacher evaluation component of the incentive model drew on existing research in the field; the Danielson model (1996) served program developers as a resource for defining teaching competencies at each level of performance. At each site, teachers were trained and rated using the Danielson framework, and this formed the accountability component of the model. The point-of-entry at the schools was the school leadership team. The model specified detailed roles and responsibilities for the principal, who appointed master and mentor teachers. These master and mentor teachers were often veteran teachers having strong pedagogical skills and credibility among their colleagues. Master teachers provided instructional leadership for their clusters (or interdisciplinary teams of teachers), and they were key implementers of the incentive-based model. As such, master teachers were released from all classroom teaching responsibilities. Mentor teachers may or may not have had released-time, depending on resources at the site; they were expected to help career teachers (or regular, full-time classroom instructors) implement new instructional strategies in their classrooms. External support staff members were dispatched periodically from the national office to assist the school leadership team in implementing the program’s teacher observation rubric, a detailed 19-point, numeric scale for observing and rating the quality of teaching. Classroom teachers were observed and rated four times a year using this rubric. Positive ratings for teachers and school leaders were converted into to cash rewards. The financial incentives also included salary raises for master and mentor teachers. As we discuss below, career teachers earned payouts based on a combination of the scores on the observation rubric, as well as value-added scores derived from the student standardized testing outcomes. Principals also received cash payouts when there was evidence of school-wide improvement.


The costs of this incentive-based model were borne by the grant, although there was cost-sharing picked up by the school district and the 12 charter school sites. The amount of cost sharing by the schools was expected to increase every year such that the incentive model could be made sustainable by the end of the grant period. The actual cost estimates were also identified as part of the implementation experience. First, there was an amount that went towards covering master and mentor teachers for release time from the classroom. At the sites we studied, this amounted to $300 per pupil, and it was covered partly by the U.S. Department of Education grant which the school district received, and the school absorbed the remaining cost. Second, there was a salary augmentation of $10,000 for the master teachers and $5,000 for mentor teachers for the duration of the study, and these were funded from the grant. This salary augmentation was in addition to regular salaries, and the extra salary came with an expectation of additional work days during the summer for training purposes. Third, there was a performance-based award pool that was available in the following maximum amounts: $750 for teaching assistants, $2,300 for teachers, and $4,000 for principals. These awards were typically based on a combination of observation ratings and student performance as measured by test scores.


At the beginning of our evaluation in 2008, this incentive-based model had already been implemented in approximately 130 schools in 14 states. In a study of this incentive model in New York City schools, Marsch et al. (2011) found that the implementation had mixed results in creating the conditions that foster success. Although teachers reported being aware of the program and were generally supportive of it, more than a third did not understand key elements of the program, including targets, bonus amounts, and how committees decided on distribution plans. The vast majority of teachers suggested that they had not been informed about distribution plans at the start of the year; felt that bonus criteria relied too heavily on test scores which in turn indicated limited buy-in for program performance measures. Teachers also seemed to overestimate the likelihood that their schools would receive an award. And although the majority of teachers expressed a strong desire for their schools to win the award, many recipients reported that, after taxes, the amount seemed insignificant. Each of these factors could have affected the program’s success. Jeffrey et al. (2014) also found that fewer than half of districts reported implementing all required components of an incentive based program. Although 85% of districts implementing this program reported at least three out of the four required components for teachers, 46% reported implementing all four. In addition, many educators misunderstood the performance measures and the pay-for performance bonuses used for Teacher Incentive Fund (TIF) grants. For example, the measures that educators indicated were used to evaluate their performance sometimes differed from those reported by districts. In addition, more than half of teachers did not know they were eligible for pay-for-performance bonuses, and teachers reported a maximum pay-for performance bonus that was lower than the amount reported by districts. According to Jeffrey et al. (2014), most teachers and principals are satisfied with their professional opportunities, school environment, and the TIF program. About two-thirds of teachers were satisfied with their jobs overall and were glad to be participating in the TIF program.


GAP IN THE LITERATURE AND RATIONALE FOR THE STUDY:


Based on the literature above it is clear that both the design of the incentive model, funding amounts, as well as, effectiveness of implementation need to be considered in order to make an incentive-based model work. While the extant research addresses key issues such as the challenges of implementing incentive-based systems (Carr, 2008; Goldhaber et al., 2008), incentive pay and student outcomes (Eberts et al., 2002), and the effects of professional development on teacher quality (Desimone, Porter, Garet, Yoon, & Birman, 2002), less is known about how teachers and school staff make sense of the introduction of new models, if/how these models become accepted and viable in schools; and how these perceptions, understanding of the model and impact of incentives unfold over time in participating sites. The studies cited thus far have predominantly studied only about 1 year of implementation mainly in public school systems. In addition, most programs mainly focused on specific aspects of a program (e.g., financial payouts based on student test scores or teacher perceptions) and do not address teacher evaluations professional development, transparency in measures, accountability and growth in student achievement in a comprehensive program model of incentive based reforms. In fact, previous failures of incentive models were attributed to the fact that the incentive model was not comprehensive enough and did not have adequate accountability systems in place. Some focused on teacher professional development, others on financial payouts, and yet others on measures of student achievement. Most also didn’t have systems in place for monitoring implementation, ensuring validity of teacher evaluations along with ensuring transparency and fairness in all aspects of the incentive model.  


The idea underlying the program that we studied was that if the model is comprehensive enough, that it would then address many of the previously cited shortcomings. There is a gap in knowledge on the processes of implementation, perceptions of incentive-based models and how they unfold over time in the school setting. We argue that even with flawless implementation of a program there are contextual forces that affect outcomes. It is in this context that we examined the implementation of a comprehensive incentive-based model that was designed to address many of the criticisms of the former approaches to performance-based incentives cited in the literature above. Few studies have examined the process of implementation over time, perceptions of the incentive-based model and how schools integrate these models into their existing modes of functioning. In the present study, we examined the implementation process of an incentive-based program in 12 charter schools in a large urban school district. Since the program design attempted to address previous criticisms of incentive-based reforms, the main purpose of the present study was to examine the viability and effectiveness of the incentive model through its implementation and to examine its evolving impact on the educators and students at the schools. As a result, in this paper, we speak to both the issues related to the design of the incentive model as well as other challenges related to implementation in the sample of schools.


METHODOLOGY


The setting for this study was a large urban school district in the Mid-Atlantic region that had been awarded a U.S. Department of Education, Teacher Incentive Fund (TIF) grant from 2007–2011. The purpose of the teacher incentive fund grant was to develop and evaluate innovative strategies for differentiating pay for teachers and administrators. Expected outcomes for new incentive-based models were bolstering teacher instructional quality and increasing student achievement. This paper emerged from the larger evaluation of the implementation and effects of the grant within the school district we studied. The research question guiding our larger analysis was examining the viability and implementation of the incentive-based school reform model. As mentioned above, the study included data from 12 charter schools that participated in a federally funded national incentive-based program over the 4-year period. The participating schools were charter schools, kindergarten to grade 8, managed by neighborhood-based establishments. The full evaluation project for this local site employed a mixed methods design which included a comparative case study and quasi-experimental design, incorporating both quantitative and qualitative analyses. Comparative case studies require bounded events (Merriam, 1998), and the boundaries of this event are the experiences of administrators and teachers at these 12 charter schools. Moreover, case study research is rooted in social context (Yin, 2003), and the context weighed heavily on the way in which implementation experiences unraveled from school to school. An advantage of mixed methods research is its methodological pluralism which yields superior research, compared to traditional mono-method research (Johnson & Onwuegbuzie, 2004). The authors served as coleaders of a team of external evaluation researchers examining the implementation and impacts of the incentive-based program over the course of the grant.


The first year of the grant was a planning year which involved deciding on the implementation design, budget allocations, obtaining appropriate clearances and approvals from Institutional Review Boards, establishing subcontracts with partners (e.g., incentive model developers, program evaluators, participating schools etc.) providing initial training and orientation, and obtaining baseline data. In this first year of implementation, five schools (Cohort 1) were recruited, but one school dropped out of the program citing limitations in implementation capacity. We removed this school from the analysis as well, as there was limited programmatic activity at the school. In the second year of implementation eight more schools (Cohort 2) were recruited. The initial study sites were secure, with four schools beginning implementation in fall 2007 constituting Cohort 1, and an additional eight schools beginning in fall 2008 forming Cohort 2, for a total of 12 schools. Cohort 2 schools included three bilingual English-Spanish schools that were part of a National Hispanic development organization. Given the gap between planning, data collection, and implementation, the research team spent 3 years studying Cohort 1 schools and 2 years studying Cohort 2 schools. The research team had ongoing access to each school throughout the evaluation; we collected and analyzed data on all aspects of the program implementation throughout the school years. In our role as evaluators, we obtained appropriate clearances from the Institutional Review Board at our home institution, as well as the school district and the local boards at each of the charter schools.


The data collection instruments for this multimethod evaluation study were developed during the first year. These included feedback surveys and interview protocols. The feedback surveys and interview protocols were initially refined to account for differences in respondent roles without further changes during the rest of the study. Qualitative data elements included semistructured interviews with school-based personnel, program staff, and selected consultants to the program from the nonprofit organization that designed the incentive-based model used for all of the federally funded multisite research participants. Interviews were conducted with a purposeful sample of teachers that included members of the leadership team (principals, master teachers, and mentor teachers) and career teachers (including new and veteran teachers). The research team collectively spent over 2 to 3 days per week for 4 years in the field over the course of the study. Our fieldwork included observations of school-based support, cluster group, and leadership team meetings and professional development sessions.


Quantitative data elements included the official classroom teacher observation scores, and teacher attitudes survey data from the incentive-based model developer (which included perceptions of school context). In addition, we collected standardized student achievement scores from state and district assessment programs and our own researcher-developed surveys of teacher and administrator perceptions. Because the evaluation was nonexperimental and therefore not designed to identify causal linkages, we used descriptive statistics and bivariate procedures to analyze quantitative data. The research team administered surveys to the leadership team at each participating school. Survey instruments typically were given out after professional development sessions that were held during the summer as well as at the end of the academic year. The data were first summarized using descriptive statistics and compared with the baseline data from the first year of implementation. In addition, we obtained our local site data from the teacher attitudes survey that the nonprofit organization that designed the incentive-based model administered across all of the federally funded sites. Interview data were transcribed and coded using a comprehensive coding guide based on the program structure and goals, then combined with emerging thematic elements in the participants’ experiences (Miles & Huberman, 1999). Observational field notes (handwritten and typed up as word documents) were maintained by the research team and analyzed as a qualitative data source. The researchers observed all events discussed above including cluster meetings, training sessions for school leadership teams, as well as school-based monitoring and reviews performed by national program staff. Systematic analyses of coded transcripts and observation notes indicated emergent themes related to distinct patterns of roles and experiences. To verify these patterns, the analysis proceeded by constant comparative analyses, in which preliminary coding schemes were developed and then revised during analysis (Glaser & Strauss, 1967). Particular attention was paid to “recurring regularities” (Guba, 1978) or salient themes that emerged from the data. We used Atlas.ti to manage and analyze interview and observational data. All members of the research team coded the qualitative data, and samples of qualitative data were cross-checked by a second coder. Although no specific reliability data were collected, themes were identified through extensive discussion within the research team.


Finally, we examined descriptive data on student achievement at each site during implementation of the model. Here, we summarized statistics by school and by overall performance of the participating sample with matched-comparison schools elsewhere in the same district. These data sources together helped us map the range of responses to the incentive-based model across all levels of staff at the schools and also helped triangulate the data sources mentioned above.

 

RESULTS


This section describes the processes of and lessons learned during the implementation of the incentive-based program. The findings depict how the program was fraught with challenges related to the context of charter schools, fidelity of implementation, perception of financial incentives, role of professional development, the role of the leadership team, and sustainability of the program beyond the grant years. Note that quotes in the text use pseudonyms for both the individuals and the schools.


THE CONTEXT OF CHARTER SCHOOL PARTICIPATION


When the school district won the U.S. Department of Education Teacher Incentive Fund grant, the district staff contracted the national program developers of the incentive-based model to implement its reform model at a cluster of public schools. District leaders attempted to recruit a pool of regular public elementary and middle schools; however, this effort was unsuccessful. It was widely believed that the local teachers’ union, although initially supportive of the grant proposal, determined later that the model was an attempt to subvert collective bargaining and conventional compensation practices. As a result, they withdrew support and blocked implementation in the city’s public schools. At that time, there were concerns that the grant would need to be returned to the federal government because of the inability to identify participating schools for implementation. However, these initial problems were not altogether surprising, as school districts with strong teachers’ unions are likely to successfully resist merit-based reforms (Goldhaber et al., 2008). Ultimately, despite tacit commitments during the grant proposal phase, the teachers’ union withdrew its support and no district schools participated in the program.


However, the school district which was the setting for this research had a growing assemblage of charter schools. District leaders approached several charter schools, and by the summer of 2007, the first set of five charter schools were recruited and began preparing to implement the incentive-based program. As mentioned above, this group constituted Cohort 1 of the study, and the schools received an abbreviated version of the incentive-based model through a Summer Institute, an introductory intensive training seminar on the professional development and financial payout components. However, problems emerged almost immediately as one of the five schools dropped out of the program by early fall 2007. The official reason was a lack of capacity to fully implement the various components of the comprehensive incentive-based model which did not align with the school’s priorities. For example, the school leadership explained that the requirement that the incentive-based program conduct four observations per teacher, per year was not practical given the school’s primary goals of being an experiential learning community. Another school requested additional time to become more familiar with the program and chose to start implementation in January 2008. A third school indicated that it took them a few months after the start of the school year to fully engage with the program due to several contextual factors in their school including an unexpected leadership change. Of the four schools that completed the program in the first year of implementation, three agreed to continue the program in year 2. In year 2 (fall 2008), eight more charter schools were successfully recruited. This was viewed as evidence of success by district leadership and the incentive-based model developer. The eight new sites, Cohort 2, made a strong commitment to the implementation of the incentive-based program and remained involved through the remainder of the grant. In year 3, all 12 schools from both cohorts sustained their participation.


Several aspects of implementation were problematic at the charter school level, as the charter school administrative structure was not similar to public schools, and many decisions about assigning master and mentor teachers were made by the school board, not the principal. As a result, not all positions were based on merit. This in turn affected the morale of the teaching staff and perceptions of the transparency of the program. In addition, the charter school teachers on average had much lower base salaries than those in the district’s public schools; thus, the financial incentives based on student achievement had limited impact, as teachers left to join schools paying higher salaries as soon as they had the opportunity. There also was some concern about continued participation beyond year 3; in order to comply with requirements of the grant, schools had to raise matching funds (in increasing proportions every year) to pay for the performance-based incentives on their own. As this matching amount increased incrementally every year, schools had difficulty sustaining their participation. Eventually they began steadily dropping out of the program over time because they could not absorb the added costs. In year 4, the final year of the program, 7 of the 12 schools dropped out (two from Cohort 1 and six from Cohort 2). The major reasons cited for the attrition were: (1) the financial challenges of raising matching funds; (2) continued difficulty implementing the various components of the complex program; and (3) changing priorities of the school to align with district, state, and federal requirements (e.g., the need to show Adequate Yearly Progress to win renewal of their charter contracts). Some school-level personnel we interviewed felt that the program was more complicated and difficult to implement than expected and did not provide the immediate student achievement results the schools had hoped for. Conversely, district-level staff and program developers attributed the difficulties to high staff turnover and the differences in support from school boards and administration.


FIDELITY OF IMPLEMENTATION


We found that a major challenge for school-based staff was maintaining fidelity to the incentive-based model during the implementation process. Fidelity was operationalized as the central features of the model being adopted as envisioned by the incentive-based model program developers. Maintaining fidelity was a dynamic process which involved detailed record keeping of implementation incentive-based program activities. These included: monthly professional development; four observations and evaluations of teachers per year that included one by the administrator; and weekly cluster group sessions led by master teachers and supported by mentor teachers. Despite ongoing professional development provided by district staff on the incentive-based model for the school leadership teams at the participating schools, there were several persistent challenges in fidelity, including some leadership team members feeling ill-equipped to transfer training about this incentive-based model to their teacher colleagues. In addition, the tools and rubrics for teacher evaluations were considered cumbersome and time intensive, and there was ongoing confusion about payout structures. As mentioned previously, the incentive-based model used a “train-the trainer” approach to professional development and implementation. We suggest that some of the difficulty could be attributed to the transfer of information between the leadership team and the career teachers at the schools. Although the program staff-led training for the school leadership teams was well received, they struggled to implement the learning in their own schools. Furthermore master and mentor teachers struggled to stay organized, maintain their schedules, complete the daily paperwork, and meet all deadlines—especially when they received inconsistent support from their respective school boards and administrators. Moreover since implementation relied on a train-the-trainer model, there was some resentment from career teachers who questioned whether it was adequate for the school leadership alone to receive the training from program, without including all staff with responsibilities for doing the work. To illustrate this point, one of the school’s career teachers reflected on the lack of clarity and misinformation about program implementation that led to confusion among teachers in her school:


Well, some of us had Ms. Cruz as a master teacher and others had another person. The people I feel like that dealt with Ms. Cruz had a clear understanding of the program. So half of us were in the dark and half of us were getting some information. And then what would happen is the ones that had the information would pass it on to the ones that didn't, but it was wrong. So it was just constant confusion; no one really had a clear idea of what was happening…it was just chaotic. (Ms. Smith, Bethune Charter School)


A career teacher at the same school further highlighted some of the implementation challenges that arose from a poor understanding of the program.


I wasn't very familiar with the rubric for a while. I wasn't clear that there was supposed to be a coaching element that was supposed to be constant. I think, too, our cluster meetings got into the clarification, which was supposed to happen in the beginning of the year, but now it's March or February and we're still talking about what the incentive model is. (Ms. Orlando, Bethune Charter School)


In response to these types of comments, the district-level staff responsible for managing the implementation of the program in all 12 schools hosted a program wide training for all teachers from the participating schools in year 3 of the program. District staff and the incentive-based program developers were pleased that more than 400 teachers attended this professional development event. It was seen as an opportunity to dramatically increase fidelity to the model, as each school in both cohorts were well represented and trained at the same time and in the same place. High teacher turnover, unfortunately, was persistent over the course of the study and common to the context of the charter schools; therefore, the impact of this kind of training did not translate significantly to the study’s subsequent years.


THE STRUCTURE OF PAYOUTS FOR PERFORMANCE


Each of the schools in the incentive-based program received payouts in 2 or more years of participation the program. The payouts were based on a complex formula, which took into account numeric scores of classroom evaluations, along with student achievement data. Table 1 outlines the payout structure for teachers and school leaders participating in the program.


Table 1. Payout Structure for Teachers and School Leaders Participating in the Incentive-based Program

Type of Program Participant

 

Payout Levels

Max. Amount

Individual

Grade Level

School-wide Growth

Teachers in tested grades and subjects (grades 3–8;

math & reading)

$2,000

50% classroom observation

30%

Value-added achievement

20%

Teachers in nontested classrooms

$2,000

50% classroom observation

0%

50%

Principals

$8,000

Teacher reports (max $4000)

0%

50% school-wide student growth (max $4000)

 

As depicted in Table 1, the maximum payment for any teacher was $2,000, while principals could earn an additional $8,000 in payouts. We learned through the observations and interviews that the pay matrix and this structure of payouts were consistently poorly understood throughout the program. Common misunderstandings involved both the amounts of money teachers could earn as well as the timing of the payments. This was surprising to the research team, as the underlining theory of action for the incentive-based model was that cash incentives would motivate staff to participate in innovative professional development intended to improve their pedagogy. Figure 1 shows teachers’ responses to survey items pertaining to their ability to explain how the payout systems work. The survey was administered to staff in both cohorts during each of the study years, 2008–2011. The news here was not all bad, however. As presented in the chart, participants seemed to steadily have more clarity on the financial incentives in all years of the program, and by the final year, 66% of respondents reported understanding the payout system.


Figure 1. Participants’ ability to explain the performance-based compensation system [39_20450.htm_g/00002.jpg]


Source: Incentive Program model survey, 2008, 2009, 2010, 2011


We also found that this payout structure had the unwanted side effect of putting extra pressure on the teachers in the tested grades to boost test scores, as school-wide payouts for everyone, all teachers and administrators, depended on students in grades 3 to 8 demonstrating measurable growth in math and reading scores during the annual state-based testing.

 

PERCEPTION OF FINANCIAL INCENTIVES


In surveys and interviews, career teachers agreed with the program philosophy on the need to reward teachers according to their differing levels of commitment and effort. However, there was widespread ambivalence and skepticism about whether cash payouts, as structured by the program could motivate a given teacher to teach better. Many of the teachers we interviewed believed that the motivation to teach was mostly intrinsic—the internal satisfaction of doing one’s job well. Yet they appreciated the financial rewards they stood to receive with the acknowledgment that standardized tests did not capture the entirety of their challenges and the multitude of factors that affected student performance. For example, when asked about her motivations for teaching, one of the career teachers said:


I love it because I feel like teachers that don't do anything are the reason why we're failing so miserably. So, that's my opinion. I think some teachers feel that way, when they do their job and they're commended for it, it makes them feel good. It's a positive thing. So I don't, I didn't find a problem with it. It was a nice reward. (Ms. Allicot, Ujama Academy)


This sentiment was echoed by another career teacher who was appreciative of the recognition and rewards for good teaching but did not view this as a key motivator.


I'm not a teacher that is in it for the money. I mean, I would like to get paid, but a payout is not what I'm doing it for. So to me, personally, the payout is not the issue. The payout for me is whether my students show growth. But I like the payout. I'm not going to lie. When the check comes, I do not deny it, I accept it. (Mr. Hayes, Ujama Academy)


Mr. Hayes eventually accepted the idea that the incentive-based program was worthwhile, especially in differentiating teacher pay to correspond with productivity. He later added,


I do believe that in some cases, merit pay is worth it because you have teachers that do phenomenal jobs and then you have teachers that at 4:00, they're ready to clock out and they're doing just the basic minimum. Yes, I believe that those teachers that are doing the phenomenal job, regardless of whether they're doing it for the pay or not, should get paid more, get a stipend just for their efforts. (Mr. Hayes, Ujama Academy)


Although several teachers appreciated the financial payout which they perceived more as a reward or bonus than a restructuring of compensation, there was limited evidence that teachers were motivated by the incentives and whether the incentives helped with teacher retention. For example, interviews and survey data indicated that the performance-based payout amount of $2,000 was not a large enough incentive to motivate significant change in teacher behaviors or retain them at the schools. Figure 2 shows the survey responses on teachers’ perceptions about the degree to which the incentive-based model could retain teachers at the school each year.


Figure 2. Participants’ perception of whether financial incentives help retain educators

[39_20450.htm_g/00004.jpg]

Source: Local site researcher-developed survey, 2009, 2010, and 2011


In interviews with the school leadership teams, we learned about a small number of teachers who disagreed with the tenets of the incentive-based model and were unwilling to participate in the program year after year. However, because the model was presented as a comprehensive, whole school reform model, teachers who were dissatisfied with the program often left the school. Others who were initially overwhelmed with the paperwork and instructional changes required for participation became more comfortable once they realized they were already implementing many of the strategies as part of their regular work responsibilities.


IMPORTANCE OF PROFESSIONAL DEVELOPMENT


A salient finding was that career teachers frequently appreciated the support and guidance from the master and mentor teachers. At sites where the incentive-based model was introduced and rolled out thoughtfully by the school leadership team, with adequate public engagement by the lead administrator, teachers were found to be more responsive. Generally, successful implementation was found to hinge on the school leaders’ ability to communicate effectively to the teachers and balance their needs with the program guidelines. Indeed, communication, support, and follow-up were particularly important for a program using a train-the-trainer approach, especially since most staff did not attend important trainings. In fact, poor top-down communication and lack of understanding about vital program components was a persistent issue throughout the incentive-based program implementation. This was attributed partly to the high teacher turnover in the schools (which meant starting training from the beginning every year) and also due in part to the complexity of the program; furthermore, the incentive-based program was often one of several ongoing initiatives in a school.


One of the mentor teachers described the challenges teachers experienced in integrating the various elements of the incentive-based model rubric into their teaching and evaluation, given the logistical challenges of the school’s schedules.


I think the biggest challenge was in the scheduling part. We were pulled out of the classroom a lot, and I always felt that once I was out of the classroom, then my children weren't getting what they were supposed to get with me out of the classroom. And it wasn't just the 45 minutes, it was 45 minutes out and then 45 minutes evaluating, then additional time writing up the evaluation, and so much more even. Go back into the room, check to make sure that what we said and what we saw was right. So it was a lot of time out, other than just doing the evaluation. (Ms. Akira, Millennium Charter School)


A school’s ability to integrate the incentive-based model into its existing mission, culture, and context as well as the leadership team’s ability to train their teachers in the program components affected implementation and eventually led decisions by several schools to drop out of the program in the third year. However, while some schools struggled and dropped out of the incentive-based program, district-level staff remained optimistic about its implementation. They believed that, while various components of the incentive-based program had differing degrees of difficulty in implementation, school-level staff had the capacity to fulfill the demands. Much of their focus and attention was placed on the instructional features of the model and on the need to help teachers to experiment and innovate with new instructional approaches. After one of the training sessions, a district staff member suggested:


The lessons are changing. Teachers are talking about the rubric as a transparent guide to help know what they should be doing in the classroom. You can tell teachers are coming together to become more collaborative. (Ms. Schwartz)


Indeed, Ms. Schwartz’s perspectives were supported by our observations and interviews, although such synergies were not consistent across schools. Several teachers we interviewed also stressed how the instructional component of the incentive-based model was not the greatest perceived need in their classrooms at first. They recounted, instead, that having difficulty with classroom management impeded their ability to innovate with their pedagogy. However, one of the career teachers stated, that his teacher mentor provided the support and guidance to simultaneously improve his classroom management skills, along with his instruction. He said:

 

I didn't start the year off on a good foot and so I was constantly having issues with behavior management and fighting in my classroom, and it was distracting for my lessons. And as the year went on, it became less about behavior management and more about my lessons, which was great. Not to say I didn't have my problems with behavior still but, but I was able to think more about the structure of my lessons and what was missing. I was able to focus with her (mentor teacher’s) help. (Mr. Rodriguez, Bethune Charter School)


A master teacher alluded to the importance of taking into account the classroom environment, including what both the teacher and student are doing, in order to conduct an effective observation. These ideas were presented and practiced during professional development sessions,


A lot of times we look at the activities and the materials that are used or what the particular lesson is about, whether it's read-specific and things like that. But, with this incentive program, we got more specific. So when I went in, I was not really looking at what the teacher was doing or how the teacher was implementing the lesson, I was also looking at how the students took it . . . did they retain the information. Because, you can do a whole pony and show dance of a lesson and then they still not get it. So it kind of trained me to look for how the students took in the lesson as opposed to how the teacher delivered it, still looking at that but more so I started to, you know, look at whether the students were mastering what was being taught. So that was valuable. (Ms. Akira, Millennium Charter School)


An interesting aspect that was reported to have increased in both the survey and interview data was collegiality among teachers, which was attributed to the time dedicated to cluster groups and professional development. This perspective was articulated well by Ms. Akira, the mentor teacher, who told us:


I think as a result of the incentive-based model we have a more collegial atmosphere. People are required to work together with clusters. I think also it made us concentrate on instruction like never before so that even our regular staff meetings become a forum for discussing and improving instruction. (Ms. Akira, Millennium Charter School)


SIGNIFICANCE OF THE LEADERSHIP TEAM


We found that identification and appointment of the leadership teams was a key issue with implications for fidelity of implementation in some schools. For the most part, master teachers and mentor teachers were handpicked by the principal or other personnel on the school’s leadership team. The more broadly represented leadership teams at the charter schools in our study included, for example, the chief executive officer, chief academic officer, or principal. The master and mentor teachers we interviewed were not completely certain why they were chosen and often had not applied for the positions. Generally, they assumed that their years of experience and credibility within the school were the basis for their selection. It was apparent from our interviews in the early years that master and mentor teachers were unclear about their job descriptions and were often unaware of program documents which explicitly defined qualifications for these positions. The master and mentor teachers we interviewed appeared to fit the roles and responsibilities of their positions, but they were not able to articulate or refer us to any job description. However, clarity on key positions within the incentive-based model changed in subsequent years when application and interview processes were established for selection of master and mentor teachers. Still, some role confusion and territorial issues persisted. For example, some mentor teachers were unsure what differentiated them from the master teachers, as the type of work they did and the expectations were similar. To add to this concern, discernment between the two roles became further clouded, as several mentor teachers were promoted to the position of master teachers, while others remained mentor teachers for the duration of the study. During the interviews, staff members who were promoted to master teachers in several of the schools expressed excitement about their new leadership role and appreciated full-time administrative work. This was viewed as a step up in authority, which included greater autonomy and control over one’s work.


In some schools, master teachers were fully released from teaching responsibility. However, several schools “doubled up” on master and mentor teachers in their schools, which meant those teachers were not able to devote as much time to the incentive-based program activities required for implementation. Despite these concerns, the interview data suggested that master teachers found more satisfaction in their roles than mentor teachers. Master teachers held a higher status in the leadership team, were not expected to teach classes, and earned larger amounts in salary augmentation. We found that mentor teachers seemed to have heavier workloads and yet received fewer rewards compared to the master teachers. Though the mentor teachers interviewed were positive about their roles, many did not express the sense of leadership indicated by the master teachers. There was generally low morale among mentor teachers. The compensation package was not an effective source of satisfaction among mentor teachers, and the work itself was grueling. This was a consistent finding throughout the study, that mentor teachers, while still having considerable teaching responsibilities of their own, found it difficult to assist in raising instructional quality throughout the school. This problem was a shortcoming of the design of the incentive-based model.


In terms of their own growth as instructional leaders, the master and mentor teachers we interviewed referred to their new opportunities for professional development and effecting instructional change in their schools as some of the benefits of participation in the program. For example, Ms. Akira discussed data-driven decision making.


We get our professional development through leadership team, so we basically look at data. We look at overall strengths and weaknesses in the school. We decide on an area that we need to work on, based on what the data says and what we see that our staff needs overall. (Ms. Akira, Millennium Charter School)


Overall, although there were inconsistencies in the appointment and capacity of the leadership teams, it seems like instructional quality improved in the schools and all members of the teaching staff received professional development.


SUSTAINABILITY


Teachers’ perceptions and concerns about the sustainability of the program, particularly funding payouts, also emerged in the interviews. Early on, teachers were concerned about the potential longevity of the compensation system, perceiving that continuation beyond the grant period was unlikely. There was talk about how the incentive-based model was simply “another project” or “flavor of the month” that would end once the federal grant funds were depleted. In order to sustain the incentive-based model beyond the grant years, participating schools were expected to share costs during the later years of implementation and to develop a robust fundraising strategy to cover the full costs at the end of the grant. While this expectation was known among all district-level staff and some members of school leadership teams, teachers were largely unaware of the ongoing fundraising component of the model. A master teacher reflected on the longevity of the program,


So what happens after these four years? I am just understanding the process as it relates to responsibility of the school with the program. Will it go away or is this in vain? I mean, personally for the individual growth, the rewards are great. But overall, that's just kind of a gray area for me because we've seen a lot of things come and go. We've seen a lot of programs. We've seen a lot of materials, what's adopted, what's taken out, whole language, phonics. When you're around a long time, we've been down this road before (Mr. Mohammed, Ujama Academy)


A district-level staff member also mirrored Mr. Mohammed’s concern for the future sustainability among schools in the program,


Several of the schools came to us and said we'd love to do this however we can't, we can't afford the match. Because next year is 25%, they'd have to match 25% (of the money for financial incentives). The following year it's 50%, and the last year is 75%. Last year we paid 100% and so schools came. But the school who dropped out may be dropping out partially because now they had to put their hand in their own pocket. Our concern is, as it gets closer to the 75%, people will drop out because they don't want to spend the money [to cover cash incentives]. (Ms. Watson, District Staff)


As mentioned earlier, the incentive-based model developers encouraged participating schools to gradually phase in a fundraising component to sustain this incentive structure. Beyond the initial struggle of recruiting sites for participation, the final years of the implementation can be viewed as a battle against attrition. Because none of the charter schools were flush with resources, they were largely unable to become financially independent to sustain the program incentives which they had created.


STUDENT ACHIEVEMENT


The end game of this incentive-based program is to improve student performance by changing the school culture and making instruction the focus of school improvement. A multivariate analysis modeling outcomes for students is beyond the scope of the present paper. However, Figures 3 and 4 show descriptive data on the pattern of math and reading achievement in the implementation schools, matched-comparison charter schools, and comparison district schools. Comparison schools were selected by matching school size, grade level, and student demographics.


Figure 3. Student math achievement data comparing implementation sites, matched comparison, and school district means

[39_20450.htm_g/00006.jpg]

Source: State Standardized Math Assessment; 2007, 2008, 2009, 2010, 2011


Figure 4. Student reading achievement data comparing implementation sites, matched comparison, and school district means

[39_20450.htm_g/00008.jpg]

Source: State Standardized Reading Assessment 2007, 2008, 2009, 2009, 2010, 2011


As can be seen from the Figures 3 and 4, there was some consistent improvement in the participating schools, although it varied from year to year. However, given all the inconsistencies in implementation and the existence of multiple initiatives in the participating schools, it was not possible to determine whether this was an effect of the program or other factors that affected all the participating charter schools.


DISCUSSION


In this paper we examined the outcomes of a comprehensive 4-year incentive-based reform model by studying its implementation in 12 charter schools in a large urban school district. The incentive-based model was designed to avoid several of the potential pitfalls of former incentive structures as identified in the literature. For example, in response to criticisms of previous models, this model provided comprehensive professional development, based payouts on multiple measures of teacher performance, provided multiple career pathways to teachers, and had a defined incentive structure. Despite the intended improvements to the model, there were several challenges to successful implementation including issues related to the context of charter schools and difficulties in understanding how to make a complex model work. The challenges to implementation in some of the schools appeared to be the result of program complexity, repeated trainings required due to high teacher turnover, limited capacity for ongoing training and support, and limited organizational support. Teachers generally reported satisfaction with the program in some schools where they had effective administrative leadership, master and mentor teachers, low turnover in leadership teams, and program buy-in. Moreover, charter school context (compared to public schools) revealed unique factors that impacted implementation. These included diversity of missions and structures, alternate and multiple leadership positions (in addition to, or in place of, principals), staff turnover (especially at the leadership level), and typically lower base salaries for teachers.


Despite efforts to account for expected weaknesses of incentive-based models, these local site program implementers were also not able to circumvent the well-known challenge of public school engagement, specifically gaining the commitment of the teachers’ union. Much of this was attributed to the policy landscape in the city and to poor communication by the program developers about the benefits and theory of action for the performance pay component of this incentive-based model. Once the charter school sites were selected, other challenges included lack of clarity and transparency in the selection of the school leadership teams, lack of fit between a school’s existing mission with the incentive-based model, and complexity of the train-the-trainer structure that prevented adequate preparation of the teachers in the individual schools to implement the model effectively. For some participating schools, although the implementation of the program had a challenging start, they were eventually able to accomplish program implementation. That is, the implementation of this incentive-based model changed the way school leaders and instructional staff approached teaching and learning and helped them to innovate and bolster student learning. Determining how much of this outcome could be attributed to the financial incentives (cash payouts) was difficult given that the program also provided professional development; in addition, there were other ongoing initiatives in the school, and each school had a different culture.


Challenges in fidelity of implementation were also related to the design of the incentive-based model itself. For example, the model required a new hierarchy in the form of leadership teams consisting of master and mentor teachers. We found that the creation of new leadership positions when not perceived as valid, led to disruptions in the social organization of the staff in some of the schools. Not all master and mentor teachers were accepted as credible leaders and not all had the capacity or experience to perform in that role. This was sometimes related to lack of transparency in the selection process, and at other times, there were pressures from the school board to assign specific people to a role. In addition, the role of the principal or head administrator was key to successful implementation. Too often weak relationships among some teacher leaders as well as administrators affected communication, perpetuated misinformation, and bred resentment. As indicated in the literature, if professional learning communities were needed to transform schools, then “professional dialogue” should have been propagated throughout the schools’ cluster and leadership team meetings, as well as pre- and post-observation conferences (DuFour, 2004).


The importance of clear communications cannot be stressed enough. Study participants were consistently unclear about the financial incentive structure—the foundational criteria for successful implementation of the program. Although our interviewees were unanimously pleased with the amounts of cash they received, they could not explain to us, for example, how the payouts would be different by teacher role and grade-level. Thus when the teachers received the financial payouts, the amounts were perceived more as bonuses or rewards for work already completed, rather than as true incentives for changing future behaviors. We found that unlike master teachers who earned significant salary augmentations (e.g., up to $10,000), the performance-based compensation amounts for the teachers were not large enough to motivate them to stay on in their schools. This is an important difference in allocation of financial incentives: master teachers and mentor teachers received salary augmentation for the professional development that they provided to the career teacher. The salary augmentation sum was not considered performance pay. The performance-based payout was given to all teachers and was based on student achievement and classroom observations of instruction. Of the three groups of teachers, master teachers were definitely more responsive to the incentives than the mentor teachers or the career teachers. This finding suggests that incentives might have been more effective if they had been tied to a percentage of salary rather than as a fixed “bonus” amount of a few thousand dollars. In addition, it is also possible that master teachers might have been most satisfied with their roles and compensation because they also enjoyed autonomy and leadership in their roles at the schools.


Over the course of participation, we found that all schools reported some growth in student scores (sometimes infinitesimal and within a margin of error) that enabled them to receive school-wide payouts. These results varied year to year with different schools demonstrating growth at different times and some even demonstrating regression. The payout components linked to student achievement were derived from value-added growth rather than aggregate standardized scores. This is especially relevant as the participating schools were all urban, low performing charter schools. The participating schools demonstrated growth in reading scores by aggregate measures, but they had not visibly fared better than their comparison schools in math. Moreover the growth in student achievement varied by year and by school, and there was no clear link between student test scores and the financial incentives.


Since, payouts for performance in the first year of the incentive-based model were based on value-added measures of student growth scores, there was tremendous pressure on the teachers in the tested grades (math and reading in grades 3 through 8) to produce measurable student growth. Since the incentives-based program was never fully implemented effectively in Cohort 1 and Cohort 2 schools, it was difficult to attribute student achievement to changes in teachers’ instruction as a result of their participation. In a report recently published by the National Academies’ National Research Council (Hout & Elliott, 2011), a team of education, law, sociology, and economics experts examined the implementation and effectiveness of various education accountability programs throughout the U.S. and abroad since 2002. These researchers determined that assessments were not used properly to determine rewards or penalties for schools, that the results of assessments and the affiliated sanctions did not change the behaviors of the educators whose behaviors needed to change, and that there were not enough ways to safeguard against manipulation of these accountability systems. In addressing this question of financial incentives, in a study of principal incentives Holly, Dean, Hassell and Hassell (2014) found that the bonus also did not appear to serve as a motivator for most principals; more than two-thirds reported that the prospect of earning a bonus did not affect their practices. Throughout the evaluation, principals consistently said that money did not motivate them to work harder in a high needs school or to change their practices to raise student achievement and that they therefore found the idea of pay for performance problematic.


An issue we found with our study was the complicated implications of incentives. Payouts were not synonymous with incentives. Incentives imply that individual performance can be changed and motivated by the promise of a future financial payment. Evidence of teachers’ future orientation was not found in our data. What we did find was that teachers and administrators often viewed the financial payouts at the end of the year to represent a “reward” for their hard work in the previous year rather than as an “incentive” for the future. Incentive implies that it motivates you to work differently and that would manifest in changes in teachers’ behavior leading to better student outcomes. Rather than being perceived as an incentive, the payouts were perceived as a bonus – an extra something at the end of the year for the hard work put in by the teacher(s). The purpose of a bonus is different from that of an incentive. We saw appreciation for the rewards, but few study participants appeared to respond to the incentives as intended. Those few who did appear to be responsive to the incentives were younger teachers or master teachers who received salary increases and leadership roles.


There are several policy implications that can be derived from this study. First, any incentive-based model should consider what it seeks to accomplish; that is, consider whether it seeks to provide incentives for improved instructional performance and/or rewards for student achievement. The design and implementation of an incentive-based model will be different based on these responses. In addition, teachers sought higher base salaries.  Small rewards that required a lot of paperwork were perceived as more of an annoyance than as an incentive. The pressure on teachers in the tested grades was very high, as the school-wide payouts depended on the scores from their classrooms. The cost and complexity of the program curtailed the sustainability of the incentive component. Policy makers might consider alternate and/or additional measures of student performances to reduce this excess pressure on a few teachers in the schools. The comprehensive structure of the incentive-based model helped to improve instruction across the schools, and teachers appreciated the time allocated to build professional learning communities and receive mentorship from an experienced teacher. These mentor roles also need to be considered carefully, as the mentorship role placed additional burdens on some teachers and not all mentors were respected or valued by the career teachers. Sustainability is also a major consideration given that schools dropped out of the program, as might have been expected, when they were unable to raise funds to pay for the financial incentives. What was sustained was the professional development which was integrated into the culture of the school. For example, schools were able to incorporate the teacher evaluation formats and the weekly professional learning community meetings.


There were several methodological challenges which should be reiterated here. The staffing at the schools was often different from year to year, thus there was often little or no continuity in study participant roles making it hard to assess actual learning and change over time. In addition, the low levels of fidelity in implementation made it very difficult to attribute any changes in teacher attitudes specifically to the incentive-based model. Teacher salaries at the charter schools varied, but overall tended to be much lower than the public schools in the district and charter school teachers had no union protections. As a result, it was difficult to determine if the small financial rewards offered by the grant had any impact on retention.  Many teachers averaged a just few years at the schools and went on to better paying jobs at other schools.


CONCLUSIONS


In conclusion, this study sheds light on the challenges of implementing a large scale, multi-year, incentive-based school reform initiative in high poverty, high-needs urban charter schools. The study findings ultimately suggest that incentive-based models, however comprehensive in theory are not yet viable and have limited effectiveness. We believe they are not viable at present because they neither provide significant financial gain, nor are easy to implement. Thus the incentive part is neither understood nor impactful. These are not temporary problems. These are fundamental problems with comprehensive incentive based models such as these. Incentive models need to provide substantial increases in teacher base salaries and provide long-term professional development support to ensure change and sustainability. While participants in the study gladly accepted additional pay as a reward/bonus for their hard work as teachers it was too small an amount to impact their instructional behaviors or retention at the school. The contribution of the study is that even when an incentive model is designed to address shortcomings of previous models, the local context of implementation is a major factor to consider. In the effort to be comprehensive, the model itself was too cumbersome making it difficult for participants to understand the model and to ensure fidelity of implementation. The key takeaway from our study is that incentivizing teachers through such a model is not effective.  Teachers need significant salary increases to be incentivized by money and the program needs to be simpler in design.


The aspect that was somewhat sustained was the professional development and collegiality among the teachers indicating that a professional community is what teachers appreciated most from the program and what realistically could be sustained from the incentive implementation experience. While payouts were appreciated, there were other priorities and values that motivated teachers to perfect their craft including commitments to teaching, and ongoing institutional supports. Future research might examine the role of different levels of financial incentives on performance and compare how the reward model works relative to the incentive-based model. Additionally, such research might also compare the role of professional community, non-financial rewards (e.g., recognition among peers, parents and students), and professional development as incentives in retaining teachers and improving student performance and the sustainability of incentive-based models.


Notes

1. The authors would like to thank our research associates and essential members of the research team, Dr. Camika Royal, Dr. Lauren Saenz, and Ms. Adele Gonzaga.

References


Andrejko, L. (2004). Value-added assessment: A view from a practitioner. Journal of Educational and Behavioral Statistics, 29, 7–9.


Anyon, J. (2005). Radical possibilities. New York, NY: Routledge.


Bryk, A. S., Harding, H., & Greenberg, S. (2012). Contextual influences on inquiries into effective teaching and their implications for improving student learning. Harvard Educational Review, 82(1), 83–106.


Carr, N. (2008). The pay-for-performance pitfall. American School Board Journal, 195(2), 38–39.


Cuban, L., & Usdan, M. (Eds.). (2003). Powerful reforms with shallow roots: Improving America’s urban schools. New York, NY: Teachers College Press.


Danielson, C. (1996). Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and Curriculum Development.


Darling-Hammond, L., Bullmaster, M. L., & Cobb, V. L. (1995). Rethinking teacher leadership through professional development schools. The Elementary School Journal, 96, 87–106.


Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational evaluation and policy analysis, 24(2), 81–112.


Dillon, N. (2008). The Merit Scale: As the focus for teacher pay shifts from experience to performance, districts are placing a new emphasis on salary reform. American School Board Journal, 195(4), 28.


DuFour, R. (2004). What is a "professional learning community?". Educational leadership, 61(8), 6–11.


Eberts, R., Hollenbeck, K., & Stone, J. (2002). Teacher performance incentives and student outcomes. The Journal of Human Resources, 37, 913–927.


Feldman, S. (2000). True merit pay. Retrieved from http://www.aft.org/presscenter/speechescolumns/wws/2000/0300.htm


Flannery, M. E., & Jehlen, A. (2008, March). Salary trends: Where is your pay plan heading? NEA Today, 26(6), 38–41.


Fruchter, N. (2007). Urban schools, public will: Making education work for all our children. New York, NY: Teachers College Press.


Gallagher, H. A. (2004). Vaughn Elementary's innovative teacher evaluation system: Are teacher evaluation scores related to growth in student achievement?. Peabody Journal of Education, 79(4), 79–107.


Gardner, D. P. (1983). A nation at risk. Washington, DC: The National Commission on Excellence in Education, US Department of Education.


Glaser, B. G., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine.


Goldhaber, D., DeArmond, M., Player, D., & Choi, H. (2008). Why do so few public school districts use merit pay? Journal of Education Finance, 33, 262–289.


Grier, T. B., & Holcombe, A. A. (2008). Mission possible: A North Carolina school district solves the problem of recruiting and retaining teachers in its most challenging schools. Educational Leadership, 65(7), 25–30.


Guba, E. G. (1978). Toward a methodology of naturalistic inquiry in educational evaluation. CSE Monograph Series in Evaluation, 8. Los Angeles: Center for the Study of Evaluation, University of California.


Hanushek, E., Heckman, J., & Neal, D. (2002). Introduction to the JHR's special issue on designing incentives to promote human capital. The Journal of Human Resources, 37, 693–927.


Holly, C., Dean, S., Hassel, E. A., & Hassel, B. C. (2014). Projected statewide impact of “Opportunity Culture” school models. Chapel Hill, NC: Author. Retrieved from http://opportunityculture.org/wp-content/uploads/2014/05/Projected_Statewide_Impact_of_Opportunity_Culture_School_Models-Public_Impact.pdf


Hout, M., & Elliot, S. (2011). Incentives and test-based accountability in education. Washington, DC: The National Academies Press.


Jeffrey, M., Constantine, J., Wellington, A., Hallgren, K., Glazerman, S., Chiang, H., & Speroni. C. (2014). Evaluation of the teacher incentive fund: Implementation and early impacts of pay-for-performance after one year (NCEE 2014-4019). Retrieved from http://files.eric.ed.gov/fulltext/ED546820.pdf


Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26.


Kozol, J. (1991). Savage inequalities. New York, NY: Harper Perennial.


Laverick, D. (2007, February). Motivation, metacognition, mentors, and money: Ingredients that support teaching expertise. Early Childhood Education Journal, 34, 247–249. doi:10.1007/s10643-006-0144-5


Lavy, V. (2007). Using performance-based pay to improve the quality of teachers. The Future of Children, 17(1), 87–109.


Leithwood, K., Tomlinson, D., & Genge, M. (1996). Transformational school leadership. In K. Leithwood (Ed.), International handbook on educational leadership (pp. 27–40). Norwell, MA: Kluwer.


Lipman, P. (2004). High stakes education: Inequality, globalization and urban school reform. New York, NY: Taylor & Francis.


Marsch, J. A., Springer, M. G., McCaffrey, D. F., Yuan, K., Epstein, S., Koppich, J., . . .  Peng, A. X. (2011). What New York City’s experiment with schoolwide performance bonuses tell us about pay for performance. (Research Brief No. 9596). Retrieved from http://www.rand.org/content/dam/rand/pubs/research_briefs/2011/RAND_RB9596.pdf


Merriam, S. B. (1998). Qualitative research and case study applications in education (2nd ed.). San Francisco, CA: Jossey-Bass.


Miles, M. B., & Huberman, A. M. (1999). Qualitative data: An expanded source book (3rd ed.). Thousand Oaks, CA: Sage.


Mirel, J. (1998). After the fall: Continuity and change in Detroit, 1981–1995. History of Education Quarterly, 38, 237–267.


Mishel, L., Allegretto, S., & Corcoran, S. (2008). The teaching penalty. Education Week, 27(35), 30.


Neild, R. C., & Balfanz, R. (2006). Unfulfilled promise: The dimensions and characteristics of Philadelphia’s dropout crisis, 2000–2005. Philadelphia, PA: Philadelphia Youth Network.


Odden, A. (2004). Lessons learned about standards-based teacher evaluation systems. Peabody Journal of Education, 79(4), 126–137. doi:10.1207/s15327930pje7904_7


Odden, A., & Kelley, C. (2002). Paying teachers for what they know and do: New and smarter compensation strategies to improve schools. Thousand Oaks, CA: Corwin/Sage.


Quaid, L. (2008, November 18). Union prez: Teacher pay tied to performance works. USA Today. Available at http://www.usatoday.com


Schlechty, P. C. (2009). Leading for learning: How to transform schools into learning organizations. San Francisco, CA: Jossey-Bass.


Silins, H., Mulford, B., Zarins, S., & Bishop, P. (2000). Leadership for organizational learning in Australian secondary schools. In K. Leithwood (Ed.), Understanding schools as intelligent systems (pp. 267–291). Stamford, CT: JAI Press.


Yin, R. K. (2003). Case study research: Design and methods (3rd ed.). Thousand Oaks, CA: Sage.




Cite This Article as: Teachers College Record Volume 118 Number 7, 2016, p. 1-34
https://www.tcrecord.org ID Number: 20450, Date Accessed: 5/26/2022 7:55:10 PM

Purchase Reprint Rights for this article or review
 
Article Tools

Related Media


Related Articles

Related Discussion
 
Post a Comment | Read All

About the Author
  • Girija Kaimal
    Drexel University
    E-mail Author
    GIRIJA KAIMAL, PhD, is an assistant professor at Drexel University. She currently leads a multi-year evaluation study on leadership development funded by the U.S. Department of Education and Lehigh University. As part of this study, the research team is examining the effectiveness of an urban leadership preparation program including how participation in the arts could ignite learning transfer to leadership practices. She is conducting research on the impact of creative self-expression on psychosocial and physiological outcomes. As part of her interest in internationalizing the context of scholars and students, she serves as a research adviser on international development projects related to gender equity and arts-based psychosocial support for vulnerable children in trauma zones.
  • Will Jordan
    Temple University
    WILL J. JORDAN is an associate professor at the Temple University College of Education. His recent scholarship has focused on conducting empirical research to enhance program and policy development for improving the conditions of education in urban schools. In recent years, he has engaged in a variety of research projects on issues such as high school and postsecondary collaborations, the connection between school leadership content knowledge in mathematics and instructional quality, and interventions to improve failing comprehensive high schools. His primary interests and expertise fall in the areas of sociology of education, urban education, high school reform, resiliency among adolescents of color, and at-risk students.
 
Member Center
In Print
This Month's Issue

Submit
EMAIL

Twitter

RSS