Profiting From Public Education: Education Management Organizations and Student Achievement
by David R. Garcia, Rebecca Barber & Alex Molnar - 2009
Background/Context: Nationally, almost a quarter of charter school students attend a school managed by a for-profit education management organization (EMO). EMOs have full executive authority over the operation and management of schools, including curriculum and instruction decisions. Because charter schools are funded with public dollars, critics have raised ethical issues associated with operating them for-profit.
Purpose/Objective/Research Question/Focus of Study: This study compares the academic achievement of EMO-managed charter schools with other charter schools and traditional public schools in Arizona. Whereas prior EMO research has focused on total scores in mathematics and reading as the academic achievement variables, this study delves further by analyzing subtest scores that distinguish between basic and complex thinking skills. We use more sensitive test data in an effort to examine the differential impact of the educational practices of EMO-managed charter schools on academic achievement.
Research Design: Student-level longitudinal test data are used for Arizona students who were enrolled in Grades 2–6 in 2001 and who remained in the same sector (EMO, non-EMO charter, or traditional public school) for the next 3 years. The test data include total scores for reading and mathematics, as well as subtest scores divided into basic and complex thinking skills. The analyses are based on a model that estimates the level of academic achievement in Year 3 using the sector of attendance as predictors and a twice-lagged achievement variable along with the other student-level covariates.
Findings/Results: For students who remained in the same sector for 3 consecutive years, attendance in non-EMO-managed charter schools had a positive effect on achievement results in total mathematics. The outcome was driven by higher scores in mathematics procedures, the basic skills subtest. For students who remained in the same sector and same school for 3 consecutive years, EMO-managed charter schools exhibited a positive effect in reading vocabulary, a basic skills subtest, and a negative effect in reading comprehension, the complex thinking subtest.
Conclusions/Recommendations: Previous research has illuminated many common teaching and learning characteristics of EMO-managed charter schools, such as drill and practice and standardized curricula that can be delivered by less experienced teaching staffs. Our results are the first empirical indication that the academic environments of EMO-managed charter schools may be associated with higher levels of academic achievement in basic skills at the expense of achievement in complex thinking skills, at least in reading. In all, the results are modest, but they deepen the available evidence about the academic impact of EMO-managed charter schools.
Education management organizations (EMOs) are for-profit companies that provide whole-school operation services to schools (Hentschke, Oschman, & Snell, 2002). Although public schools have outsourced or contracted with private providers for some time, EMOs are distinct because they have executive authority over the operation and management of schools (Miron, 2007), including decisions about curriculum and instruction.1 In recent years, EMOs have proliferated in large part because of the increase in the number of charter schools. Charter schools are funded with public dollars, so ethical issues of running schools for-profit (e.g., the potential conflict of interest built into the arrangement) have been raised (Conn, 2002). Supporters argue that the profit motive will encourage innovation, efficiency, and programmatic diversity as schools distinguish themselves to maintain student enrollment (Hentschke et al.). Opponents see profits as money diverted from the classroom that could be better spent enhancing the educational experience of students. They argue that the profit motive may divert funds away from academics and have a negative impact on student achievement (Conn; Loschert, ONeil, & Winans, 2004; Molnar, 2001).
This study compares the academic achievement of EMO-managed charter schools with other charter schools and traditional public schools in Arizona. In previous charter school research, reading and mathematics total scores are used as the academic achievement outcomes or dependent variables (Bifulco & Ladd, 2006; Booker, Gilpatric, Gronberg, & Jansen, 2004; Buddin & Zimmer, 2005; Hanushek, Kain, Rivkin, & Branch 2005; Sass, 2006; Solmon, Paark, & Garcia 2001). In addition to evaluating total achievement scores, we use student-level test scores to conduct separate analyses of subtest results for basic skills and complex thinking skills in reading and mathematics.
This more in-depth assessment of academic achievement illuminates contradictory subtrends within total scores. In cases in which either charter or EMO-managed charter schools outperformed traditional public schools, the advantages were due largely to increases in the basic skills subtests, and in one case (reading comprehension), the effect of attendance in an EMO-managed charter school was negative in the complex thinking subtest. These results enable us to glimpse inside the black box of EMOs to provide insight into the relationship between the curricular and instructional practices of EMOs and student academic achievement.
The article begins with an introduction to EMOs, including a discussion of EMO growth over the past 7 years. We then review the research on EMOs and discuss the literature on charter school academic achievement germane to our work, and present the national implications of studying Arizonas EMO-managed charter schools.
AN INTRODUCTION TO EDUCATION MANAGEMENT ORGANIZATIONS
An education management organization, or EMO, is a for-profit firm that manages a school receiving public funds, including conventional district public schools or publicly funded charter schools . EMO-managed charter schools operate under the same admissions requirements as regular public schools, meaning that they are prohibited from using selective admission policies. EMOs typically manage two types of public schools: district public schools, which EMOs may manage under a contract with a local school district, and charter schools. Depending on state law, charter holders may include academic institutions, nonprofit foundations, parents, or teachers. Charter holders frequently contract with for-profit EMOs to manage charter schools on their behalf. Less often, EMOs hold charters directly (Molnar, Wilson, & Allen, 2004). This study does not cover for-profit private schools, including those that may receive public funds under tuition voucher programs, such as those that operate in Milwaukee or Cleveland. In addition, the study does not cover charter schools run by nonprofit organizations such as the Knowledge Is Power Program (KIPP).
According to the Profiles of For-Profit Education Management Organizations (Profiles), an annual report that compiles information on for-profit EMOs that manage traditional and/or charter schools, the number of EMOs and the number of EMO-managed charter schools have nearly quadrupled over the past 7 years. In 19981999, the first year of the report, 13 EMOs operated 135 schools in 15 states. In 20052006, 51 EMOs managed 521 schools, with a total enrollment of 237,179 students across 29 states and the District of Columbia (Table 1).
Table 1. Growth of EMOs and EMO-Managed Charter Schools, 19981999 to 20052006
Note: From Profiles of For-Profit Education Management Organizations, 20042005 by A. Molnar, D. Garcia, C. Sullivan, B. McEvoy, and J. Joanou, 2005, Tempe: Arizona State University, Education Policy Studies Laboratory, Commercialism in Education Research Unit.
aIncludes the District of Columbia.
Charter schools account for a large and growing percentage of EMO contracts; in fact, 84% of the privately managed schools covered in the 20052006 Profiles report are charter schools. Nationally, EMOs enroll a large percentage of the total charter school student population nationwide. In 20052006, the 51 EMOs in the Profiles report account for 25% of all students enrolled in charter schools. A closer look at EMO enrollment counts reveals that EMO-managed charter schools represent an even larger share of elementary (K8) charter school students; 36% of all charter elementary school students are enrolled in EMO-managed charter schools (Molnar et al., 2006). These data suggest that for-profit firms have concentrated expansion efforts on elementary schools.
CHARACTERISTICS OF EMO-MANAGED CHARTER SCHOOLS
In general, for-profit EMOs operate according to a business model that includes characteristics such as larger schools, standardized curricula, inexperienced teaching staffs, and an underrepresentation of students who are more expensive to educate. On average, charter schools are smaller than traditional public schools, but EMO-managed charter schools are generally larger than other charter schools. Among the charter schools managed by large EMOs, 66% have enrollments exceeding the average U.S. charter school enrollment (Molnar et al., 2006).
EMOs tend to emphasize standardized curriculum across campuses in an effort to differentiate their schools from others through branding. Similar to branding in other corporate contexts, the primary motive of such efforts is to draw more students by providing an identifiable product. Proponents argue that the ability to differentiate schools is very attractive to parents who are tired of seeing large school districts trying to pound square pegs through round holes (Hentschke et al., 2002, p. 6). Commonly, EMO-managed charter schools emphasize a traditional or back-to-basics curricular program (Hentschke et al.; Levin, 2002). Over half of the EMOs studied by Bulkley (2002) indicated that they used structured, standardized curricula, with several citing Success for All or Direct Instruction specifically. Anderson (2005) studied a single EMO that operates more than 50 charter schools in multiple states and noted that the majority of the instruction is based on the Hirsch Core Knowledge Sequence.
At the elementary level, achievement outcomes are most readily influenced by standardized curriculum using drill-and-practice-oriented instruction (Nichols, Glass, & Berliner, 2006), leading to concerns that although these practices may improve students abilities on more basic skills, they could have a negative effect on the development of the complex thinking skills needed in the higher grades. For example, Success for All and Direct Instruction rely heavily on scripted teaching and drills to teach reading at the K6 level (Association for Direct Instruction, 2003; Success for All Foundation, 2005). The Core Knowledge curriculum provides less scripting than the mentioned programs but does include a rigid grade-based structure that dictates what students learn and when (Core Knowledge Foundation, 2007).2
Standardized curricula tend to require less experienced staff with lower levels of training, which could be an explanation for the relative inexperience of charter school teaching staffs. Harris (2006) found that teachers in Michigans charter schools earned approximately 33% less than teachers in traditional public schools, with two thirds of this difference attributable to lower levels of experience and education, and fewer certified teachers. Miron and Nelson (2002) found that the teachers in Michigans charter schools were substantially younger than the teachers in traditional public schools, with over 67% of the teachers at National Heritage Academy schools being under 30 years of age. Hess, Maranto, Milliman, and Grammatico Ferraiolo (2002) found that, on average, Arizona charter school teachers had 12 years of experience, compared with nearly 20 years of experience for traditional public school teachers.
Others have suggested that EMOs prefer to run elementary schools that are, in general, less costly, by shying away from serving the more expensive-to-educate populations such as special education students (Horn & Miron, 2000; Miron & Nelson, 2002; F. H. Nelson, Drown, Muir, & Van Meter, 2001). Henig, Holyoke, Brown, and Lacireno-Paquet (2005) found that EMOs were almost three times as likely not to offer high school grades, which they suggest is due in part to the additional expenses in terms of sports, labs, advanced placement classes, and so on. Although Lacireno-Paquet (2004) found no significant difference in the attendance of low-income or minority students in EMOs, she did note that the lack of a transportation requirement had a significantly negative effect on these subgroups attending EMO-managed charter schools. Schools that are not required by their state to provide transportation enrolled 22% fewer low-income students (as measured by free and reduced lunch eligibility) and 20% fewer minority students compared with schools in states where transportation was required.
RESEARCH ON ACADEMIC SCHIEVEMENT IN EMO-MANAGED AND OTHER CHARTER SCHOOLS
Much of the available research on academic achievement in EMO-managed charter schools has been produced by EMOs themselves. For example, Edison Schools (2006), in their most recent annual report (20052006) claimed, Edison schools are improving at rates well above local norms (p. 8). National Heritage Academies (NHA; 2007) consistently reports above-average results for their students. NHA has hired outside organizations to conduct evaluations of their schools, but the results have never been published (Hess & Leal, 2003; Wolfram, 2002). The Leona Group (2006) claims improved academic performance in Arizona, where 9 of 17 schools accomplish the federal benchmarks for Adequate Yearly Progress, and the state has classified all 17 schools as performing.
The self-produced results have not been confirmed by outside evaluations of EMO-managed schools. For example, Edison schools have been found to have either the same or slightly lower academic achievement gains compared with schools with similar student populations (Dryden, 2004; Gomez & Shay, 2000; Minneapolis Public Schools, 2000; Miron & Applegate, 2000; F. H. Nelson & Van Meter, 2003; Shay, 2000). A study commissioned by Edison found positive results after several years and fidelity to the Edison model (Gill et al., 2005).
In Michigan, the state with the highest percentage of EMO-managed charter schools, Horn and Miron (2000) found that NHA schools had smaller gains on state assessments compared with surrounding district schools. Whereas the percentage of students passing the Michigan Educational Assessment Program (MEAP) increased in host district schools, the results for EMO-managed charter schools were either stagnant or, in some cases, exhibited a downward trend. The Leona Group schools had one of the poorest performance records in Michigan (Miron & Nelson, 2002), and Mosaica schools ranked below average when compared with demographically similar districts at 9 out of 11 sites reviewed (F. H. Nelson & Van Meter, 2003).
Few studies have taken a macro perspective to examine the academic achievement of the EMO-managed charter school sector by analyzing results across companies. In a study of Michigan EMOs, Miron and Nelson (2002) reported fewer gains over time in EMO-managed charter schools than in other charter schools. In contrast, Loveless (2003) found that despite low starting points, EMO-managed charter schools were making larger gains than non-EMO schools. The U.S. Government Accountability Office (GAO; 2003) analyzed the test scores of EMO-managed charter schools in six cities and concluded that the majority of schools had negative results, but the findings overall were mixed.
The Comprehensive School Reform Quality Center (CSRQC; 2006) and the GAO (2002) have both reviewed the evidence on the academic achievement of EMO-managed charter schools using stringent selection standards and concluded that there is insufficient evidence to evaluate the effectiveness of EMOs. In the CSRQC study, only 9 of the approximately 900 studies and documents reviewed met the centers standards (CSRQC, 2006). The GAO, researching EMOs in the District of Columbia, identified only one study that met their approval.
There is a growing body of research on charter school academic achievement (Charter School Achievement Consensus Panel, 2006). The research that uses student-level longitudinal data is most germane to our work. These studies include student-level covariates and/or individual fixed effects and have incorporated key organizational characteristics of charter schools to improve the estimation of charter school effects. Thus far, results are mixed.
In Texas, two separate studies using individual fixed effects models arrive at different conclusions regarding charter school performance. Hanushek, Kain, and Rivkin examined test scores from 1996 to 2001 for Texas elementary students and found that new charter schools perform significantly worse than traditional public school students, but the achievement of charter schools that are at least 2 years old is on par with traditional public schools. Gronberg and Jansen (2001) found that charter schools tailored toward at-risk students gained slightly more than traditional public schools, whereas not-at-risk charter schools performed slightly worse than traditional public school students. There is a positive charter school effect in the second year due to a drop in test scores in the first year that students enter charter schools. Bifulco and Ladd (2006) found that the academic achievement levels of North Carolina charter schools were lower relative to traditional public schools for all charter schools regardless of vintage. The North Carolina authors attribute approximately 30% of the negative charter school effect to the deleterious effect of student mobility, or students transferring between schools.
Buddin and Zimmer (2005) used student-level data to examine California charter schools, with a focus on differential academic achievement results by key charter school characteristics. At the elementary level, new charter schools underperformed relative to established charter schools. This trend is reversed at the high school level, at which new charter schools generally outperformed established charter schools. Buddin and Zimmer also reviewed characteristics such as whether the charter is a start-up or conversion school, and whether it is classroom or non-classroom based. They found that non-classroom-based charters perform substantially worse regardless of whether the school is new or a conversion.
Sass (2006) is an exception in the literature because it includes for-profit status as an explanatory variable, which allows one to compare the differences in academic achievement between charter schools operated by for-profit and nonprofit entities. He found that profit status had no statistically significant effect on academic achievement, leading to the conclusion that charters managed by for-profit firms perform the same as those operated by nonprofit entities (p. 119).
Our research builds on the existing literature by examining academic achievement at a more granular level using the reading and mathematics subtests to examine differences in basic and complex thinking skills within total scores. In all the aforementioned studies of charter school achievement, total subject scores in reading and mathematics are the measure of academic achievement. These aggregated measures may be too gross to detect the relationship between the curricular and instructional practices of EMO-managed charter schools and academic achievement. We use the more sensitive test data to examine the extent to which the educational practices of EMO-managed charter schools have a differential impact on student academic achievement compared with other charter schools.
EMO-MANAGED CHARTER SCHOOLS IN ARIZONA
Arizona, the state with the second largest number of charter schools, is an informative locale to study EMOs because the state has the largest number of EMO-managed charter schools of any state. Of the 51 EMO companies profiled in the 20052006 Profiles of For-Profit Management Organizations, 30 operate in Arizona (Molnar et al., 2006), and most (21) of the Arizona EMOs are categorized as small, operating three or fewer schools. Arizonas diverse range of EMOs distinguishes it from other states. In Michigan, for example, a few large EMOs dominate the charter school offerings. In 2003, approximately 23% of Arizona charter schools were managed by for-profit companies (Molnar et al., 2004), and 25% of the charter school students in the state attended an EMO-managed charter school.
The Arizona charter school movement is of national significance because charter schools have flourished on a large scale, and policy makers have positioned charter schools as competitors to traditional public schools (Medler, 2004). The Center for Education Reform (2006) ranks Arizonas charter school law as the fourth strongest in the nation based on factors such as the unlimited number of schools, the wide range of charter applicants, and fiscal autonomy. As a result, Arizona is home to more charter schools than any other state, with the exception of California (Lake, & Hill, 2005). Arizona can be seen as a test case for the outcomes of other large-scale school choice experiments, such as vouchers (McEwan, 2000), and has been called first statewide free market in education (Gresham, Hess, Maranto, & Milliman, 2000, p. 751).
Arizona policy makers have encouraged the creation of charter schools with legislation and administrative procedures highly favorable to the growth of the charter school sector (Garn, 1999; Hassel, 1999). In Arizona, charter schools are considered alternatives to traditional public schools for the purpose of providing additional academic choices for parents and pupils and to improve pupil achievement (Charter schools; purpose, 1994). Arizona charter schools are independent public schools funded by public dollars and operate under a 15-year contract, or charter, with a governmental sponsoring entity to provide educational services to students. Arizonas market-based education system is governed by parental choice, and Arizona statutes do not require charter schools to serve a particular student population or to achieve a racial balance. Taken together, these factors make Arizona a laboratory of sorts for examining the academic achievement of EMO-managed charter schools and charter schools more generally.
This study makes use of a longitudinal data set containing nearly all Arizona students in elementary Grades 28 from 2001 to 2003. The database was created using student records from the Stanford Achievement Test, Ninth Edition (SAT9). According to Arizona law, all districts and charter schools are required to test all eligible students annually with the SAT9 (Charter schools; requirements, 1994). The legal obligation to test students creates a near census of Arizona students accounted for in the database. On average, 87.4% of all eligible student records, which includes students in Grades 26 in 2001, are linked from 2001 to 2003. In total, the database includes 221,837 linked student test records.
The study focuses on five separate cohorts of elementary students enrolled in Grades 26 in 2001 who attended schools in the same sector all 3 years. The sectors include traditional public schools, EMO-managed charter schools, and other charter schools. Between 2001 and 2003, students took the SAT9 three times, with two opportunities to transfer schools. The school transfer opportunities occur between adjacent school years (20012001and 20022003).3 The SAT9 is administered annually in April. Therefore, the test records capture the school of attendance during the spring of each academic year.4
The selection of test score metric is important because the metric can influence the conclusions drawn (Seltzer, Frank, & Bryk, 1994). Scale scores are in our analyses because they are linear, span all grade levels of the SAT9, and are trait referenced rather than norm referenced.
In addition to total reading and mathematics, individual analyses are conducted for basic skills and complex thinking skills. The SAT9 reading and mathematics assessments each are divided into two separate subtests; reading includes vocabulary and comprehension, and mathematics includes procedures and problem solving. Every SAT9 item is placed into one of two cognitive classification categories, basic understanding and complex thinking skills. The basic understanding, or basic skills, measure the students ability to recall or recognize factual information, to identify something that is explicitly stated and to associate relevant aspects of specific content (Harcourt Assessment, 1996, p. 5). The complex thinking skills measure the students ability to analyze and synthesize information; to classify and sequence information; to compare and contrast information; to evaluate information in order to determine cause and effect, fact and opinion, relevant and irrelevant; and to interpolate and/or extrapolate beyond information in order to draw conclusions, make predictions and hypothesize (p. 5). The reading vocabulary and mathematics procedures subtests contain mostly items that measure basic skills, and the reading comprehension and mathematics problem-solving subtests contain mostly items that measure complex thinking skills (see Tables 2 and 3 for a description of the subtests).
Table 2. Paraphrased Descriptions of the SAT9 Reading Test Objectives by Subtests
Table 3. Paraphrased Descriptions of the SAT9 Mathematics Test Objectives by Subtests
The student demographic covariates, gender, grade, race/ethnicity, English language learner (ELL) status and special education (SPED) status were obtained from the test records. Either students self-reported the information on the test answer sheet according to a standardized set of instructions, or the district office coded the appropriate information.
Schools were identified by unique identification numbers provided by the state and verified by the research team. EMO-managed status was established by comparing each charter school in the database with the 2001, 2002, and 2003 versions of the Profiles annual reports. Schools that appeared in the report were coded as EMO-managed charter schools.5
School mobility, defined as student transfers between schools, was coded using a unique school identification number. If a student attended a school with a different school identification number in adjacent years, the student was coded as having transferred or moved schools. In addition, a flag was created to identify students who remained in the same sector for all 3 years.6
The following analyses are based on a model that estimates the level of academic achievement in Year 3 using the sector of attendance as predictors, and a twice-lagged achievement variable along with the other student demographic variables as covariates.
Yit = αDit + βYi, t-2 + γXit + δS + eit
where Yit is a test score for student i in year t. Yitj-2 is the twice-lagged test score for student i in year t-2. Dit is a set of dummy variables indicating sector of attendance (charter, EMO-managed charter, with traditional public schools as the excluded group) for student i at time t. Xit is a vector of all other covariates, including student demographics and mobility. S is select interactions between school mobility variables if such interactions improve the model and estimation of sector of attendance.7
The first analysis includes students who remained in the same sector all 3 years but not necessarily in the same school. The second analysis uses the same covariates but added the limitation that the student must have remained in the same school all 3 years. Although this restriction reduces the overall number of students in the second analysis, it also provides two chief benefits. The second analysis isolates the effects of school practices on student academic achievement by including only those students who were exposed to school practices for the longest period of time. In addition, the second analysis is limited to schools in operation for at least 3 years, effectively precluding the negative effect associated with new charter schools that has been found in other studies (Hanushek et al., 2002; Sass, 2006).
A student fixed-effects model was considered based on previous literature (Bifulco & Ladd, 2006; Hanushek et al., 2005; Sass, 2006; Solmon et al., 2001; Zimmer & Buddin, 2005) and recommendations from organizations that review and evaluate research (Charter School Achievement Consensus Panel, 2006). Student fixed-effects models are used to control for the selection bias implicit in charter school attendance. Although Ballou, Teasley, and Zeidner (2006) noted that a student fixed-effects model is viable to control for those unobservable differences between charter school students and traditional public school students, they made the case that students who move across sectors may also not be truly representative of all charter school students. They find that student movers are different from the overall population of charter school students with regard to their school choices. Most striking was that the charter school effects were negative at the elementary level in the models without student fixed effects but positive with those with fixed effects.
The standard rationale for using a student fixed-effects model is to control for the selection bias that potentially exists between students who choose to enter a charter school and those who remain in a traditional public school. The source of the bias stems from the likelihood that charter school choosers are likely different from those students who remain in the traditional public school sector in ways that impact academic achievement outcomes. According to this logic, a selection bias should not interfere with our comparison of other charter schools with EMO-managed charter schools because in both instances, the students are choosers and should share similar characteristics, meaning that any unobservable qualities of choosers that can influence academic achievement is effectively held constant when comparing these two student groups relative to traditional public school students.
Three percent of Arizona students in Grades 26 in 2001 attended a charter school for 3 consecutive years, and 23% of these charter school students attended an EMO-managed charter school. The subgroup of charter school students who remained in the charter sector for 3 consecutive years represents approximately half of the statewide elementary charter school population in 20012002 .
Charter schools in general, and EMO-managed charter schools in particular, enroll a higher percentage of White students and a lower percentage of non-White students than traditional public schools. Traditional public schools enrolled 55% White students, and charter schools and EMO-managed charters enrolled 65% White students. Hispanic students, Arizonas largest student minority group, were underrepresented in charter and EMO-managed charter schools. Only 22% of students in EMO-managed charters and 15% of students in other charter schools were Hispanic, compared with 31% of students in traditional public schools. EMO-managed charter schools reported that 1.2% of their students were classified as ELL in 2001, whereas 4.1% of students in other charter schools and 14% of students in traditional public schools were classified as ELL. There were minimal differences between sectors in the percentage of special education students and students taking the test with accommodations (Table 4).
Table 4. Student Demographic and Academic Characteristics for Students in the Same Sector for the Years 20012003, by Sector of Attendance, 2001
Charter schools enrolled a younger student body than traditional public schools. More than half (57%) of students in EMO-managed charter schools were in second or third grade in 2001, compared with 47% for other charters and 39% for traditional public schools. By comparison, the percentage of students attending traditional public schools is evenly distributed across grades. The overrepresentation of students in the lowest grades supports the assertion in previous research that EMO-managed charter schools target the lower grades (Molnar et al., 2006; Molnar, Garcia, Sullivan, McEvoy, & Joanou, 2005).
Students attending both EMO-managed and other charter schools are less mobile than traditional public school students. Whereas 76% of charter school students and 79% of EMO-managed charter students remained in the same school from 2001 to 2003, only 54% of traditional public school students remained in the same school for the 3 years under study.
ANALYSIS 1SAME SECTOR
Table 5 contains the results of separate analyses for total reading and total math scores, the basic skills subtests (reading vocabulary and mathematics procedures) and the complex thinking skills subtests (reading comprehension and mathematics problem solving). For ease of presentation, Table 5 includes only the coefficients of interest; the full results are available in Appendix Table A1.
Table 5. Select Regression Coefficients, Students in Same Sector from 2001 to 2003
In total reading, the effect of attending a charter school or EMO-managed charter school was statistically insignificant for the total score and most subtests. Students in EMO-managed charter schools however, had a statistically significant 2.3 scale score advantage, on average, in reading vocabulary compared with traditional public schools.
In total mathematics, effect of charter school attendance was positive and statistically significant in total mathematics and the more basic-skills oriented mathematics procedures subtest. On average, charter school students scored 0.9 and 1.5 scale score points higher on the 2003 test than did traditional public students after accounting for student demographic covariates. The results for EMO-managed charter schools were statistically insignificant.
On nearly every scale and subscale analyzed, moving schools has a detrimental effect on academic achievement. In particular, moving schools has a substantially larger impact in reading, with the effect of a single school move ranging from - 6.8 to - 4.9 scale score points. The negative impact of a single school move was sufficient to eclipse the sector effects.
Finally, it mattered when students moved schools. In most cases, test scores were particularly affected by moves in the earlier grades, with a positive coefficient indicating that later school moves had a more positive impact than earlier moves.
ANALYSIS 2SAME SECTOR AND SAME SCHOOL
This analysis is limited to those students who remained in the same school from 2001 to 2003. Approximately 54% (113,143) of the traditional public school students, 76% of nonprofit charter students, and 79% EMO-managed charter students remained in the same sector and school for all 3 years, reflecting the increased prevalence of the K8 model in the charter and EMO sectors. As expected, fewer students were eligible for this analysis, but the student demographic and academic characteristics across the sectors remained similar to the previous analysis. The descriptive statistics table is available in Appendix Table A3.8
Table 6 contains the results of separate analyses for total scores in reading and mathematics, the basic skills subtests (reading vocabulary and mathematics procedures) and the complex thinking skills subtests (reading comprehension and mathematics problem solving). For ease of presentation, Table 6 includes only the coefficients of interest, and the full results are available in Appendix Table A2.
Table 6. Select Regression Coefficients, Students in Same Sector and Same School from 2001 to 2003
In reading, the effect of attendance at an EMO-managed charter school on total scores was statistically insignificant. The nonsignificant results in the total score, however, mask admonitory trends in the subtest scores. The EMO-managed charter school effect was positive and statistically significant in the basic skills area of reading vocabulary (2.9 scale score points). Simultaneously, the EMO-managed charter school effect was negative and statistically insignificant in the complex thinking skills area of reading comprehension (- 1.6 scale score points). The effect of attendance in another charter school exhibited a similar pattern to the EMO-managed charter schoolspositive effects in basic skills and negative effects in complex thinking skillsbut the results were not statistically significant.
Attending either type of charter school had a nonsignificant effect on mathematics total scores and the subtests.
Much of the existing literature on charter school academic achievement has come to mixed conclusions (Bifulco & Ladd, 2006; Buddin & Zimmer, 2005; Gronberg & Jansen, 2001; Hanushek et al., 2002; Sass, 2006). Overwhelmingly, this research has used total test score as the measure of academic achievement. In an effort to better understand the mixed conclusions, researchers have turned to other organizational characteristics, such as the age of the charter school, chartering agency, conversion status, and whether the school targets a particular student population, to provide further insight.
This study examines differences in academic achievement among schools that are defined by another organization characteristic, management by an educational management organization, or EMO. We approached this inquiry, however, using subtest scores as the measure of academic achievement. The subtest scores allowed us to differentiate the academic achievement results by basic versus complex thinking skills. We found admonitory subtrends within total scores in which charter and EMO-managed charter schools exhibited higher achievement results in basic skills areas and, in some cases, negative effects in complex thinking skills. Our results have implications for policy considerations on the operation of schools by EMOs and for future charter school research.
The first analysis consisted of students who remained in the same sector for 3 consecutive years. Neither charter schools nor EMO-managed charter schools exhibited a consistent advantage in academic achievement relative to traditional public schools in total scores, with the lone exception of non-EMO-managed charter schools that exhibited higher achievement levels in total mathematics relative to traditional public schools. The total score results are consistent with previous research that has found no statistically significant differences in the academic achievement based on for-profit status alone (Sass, 2006). Within total scores, however, charter schools had small positive and statistically significant results in mathematics procedures, and EMO-managed charter schools had a modest statistically significant advantage in reading vocabulary. Both mathematics procedures and reading vocabulary are categorized as basic skills subtests.
In the second analysis, we focused on students who remained in the same sector and same school for 3 consecutive years. This analysis is concentrated on those students exposed to EMO-managed charter schools for a substantial amount of time and eliminates the effect of test score dips associated with moving schools. Under these conditions, EMO-managed charter schools exhibited a modest positive and statistically significant effect in reading vocabulary, a basic skills subtest. Simultaneously, the effect of attendance in an EMO-managed charter school was small and negative in reading comprehension, the more complex thinking subtest.
Previous research has illuminated many common teaching and learning characteristics of EMO-managed charter schools. Others have observed that EMOs prefer drill and practice, and standardized curricula that can be delivered by less experienced teaching staffs (Anderson, 2005; Bulkley, 2002). Our results are the first empirical indication that the academic environments of EMO-managed charter schools may be associated with higher levels of academic achievement in basic skills at the expense of achievement in complex thinking skills, at least in reading.
As profit-seeking enterprises, EMOs respond to the incentives provided by the legal and financial systems in which they operate. The bottom line is that for-profit schools must spend less than they collect. In so doing, EMO-managed charter schools face an internal conflict between cost savings and quality of education. As for-profit ventures with an incentive to lower costs, EMOs make choices that have immediate and special education significance, such as choices related to . . . the quality and quantity of educational resources available to students (Conn, 2002, p. 144). EMOs weigh such decisions in light of the inherent contradictions of raising educational productivity. Education is labor intensive, with limited ability to change production through investments that reduce the need for skilled labor (Levin, 2006, p. 171). Furthermore, other strategies, such as taking advantage of economies of scale or implementing new practices, have proved discouraging.
In all, the results are modest, but they deepen our understanding of the available evidence about the academic impact of EMO-managed charter schools. Much of the previous research has found mixed evidence of differences between the academic achievement of EMO-managed charter schools and other charter schools. Detailed analyses of student-level academic achievement data, like the ones provided here, may be necessary to fully capture the impact of the educational choices made by EMOs. Our work establishes a link between the common educational strategies of EMOs, such as standardized curricula and less experienced teachers, and increased performance on basic skills.
Under No Child Left Behind (NCLB) and in most other research, total scores are the chosen metric of academic achievement; the analysis of total scores alone conceals differences by type of skill. Our research demonstrates the advantage of using subtest scores in addition to total scores when conducting research on schools generally, and on charter schools in particular. Total scores can show in significant results, whereas troubling differences between traditional public schools and charter schools can be hidden below the surface. Linn (2000) noted that subtests can provide a glimpse into how schools interpret content standards and the curricular choices that they then make. Given the findings reported here, we advocate for the increased use of subtest scores in addition to differences in academic achievement by schools organizational characteristics as a fruitful approach in future research.
Future research on EMOs could delve more deeply into organizational characteristics to further explain differences in academic achievement. For example, Bulkley (2002) defined three levels of control, both of administration and curriculum, exerted by EMOs over the charter schools with which they are affiliated and found considerable differences in the approach to curriculum, assessment, and instruction between EMOs in the three levels. High-influence EMOs preferred a standardized curriculum, centrally prescribed assessments, and a specific instructional approach. In contrast, all-purpose and mixed-autonomy EMOs allowed the school to influence the specific curriculum and assessments used, with varying degrees of control over instructional methods. Differentiating EMOs based on the level of control they exert over the schools they manage might be a way to tease apart the impact of the for-profit business model itself from the curricular approach that the business chooses.
Maranto and Maranto (2006) argued that the corporate delivery model of education can have the greatest impact on public education in part because corporations accept the legitimacy of standardized testing (p. 163). The acceptance of test scores as a means of evaluating school performance should be balanced with a commitment to transparency. Parents and policy makers need more than the total scores now reported to the general public to get a comprehensive picture of academic performance. The reporting and analysis of subtest scores should be considered in order to safeguard against the possibility of raising student achievement on the more basic skills items, which in turn can boost total scores, with the possible contrary result of impeding the development of more complex-thinking skills.
1 Our focus on EMOs as for-profit companies is not intended to overlook the fact that public school districts routinely enter into contracts for goods and services that bring a profit for the private companies that enter into such contracts. EMOs are a distinct class, however, because for-profit companies are contracted to operate or manage the core teaching and leaning activities of a school.
2 Some researchers have found that drill-and-practice, also referred to as teacher-centered instructional techniques, can be effective for disadvantaged children. See Chall (2000) or Matczynski, Rogus, Lasley, and Joseph (2000) for further information
3 In addition, individual student records may be broken because student scores were missing in any year of the study.
4 This method underestimates the degree of student mobility because students may move more than one time between test administrations.
5 There were a small number of cases in which a school converted from for-profit to not-for-profit status during the years 20012003. In those cases, the school was coded as a not-for-profit school. This coding change affected fewer than 30 students.
6 Based upon the suggestion by C. Nelson and Hollenbeck (2001), we searched for available data on teacher experience and education, as well as other school-level variables. It was found that the data quality for these variables was poor because of missing or improperly coded data, so these data were not included in the analyses.
7 A gain model, with the difference between 2003 and 2001 scale scores as the dependent variable, also was estimated. There were no significant findings in this model, and it is not reported.
8 The key difference was in the public school sector, where the number of fifth-grade students who stayed in the same school all 3 years dropped to a small fraction of those eligible because of the transition to middle school. In the initial data set of students who remained in the same sector all 3 years but who could have changed schools within that sector, we found approximately equal percentages of students in each grade of the traditional public schools (~ 20%). In contrast, the percentage of traditional public school students in fifth grade who stayed in the same sector and same school all 3 years was only 6%. The number of students dropped as a result of the structural transition from elementary to middle school that occurs after either fifth or sixth grade in most public school systems. In contrast, a large percentage of charter schools are K8, meaning that fifth-grade students can remain in the schools for another 3 years.
Anderson, A. L. (2005). The charter school initiative as a case of back to the future. Educational Foundations, 19(1/2), 3350.
Arizona Department of Education. (2002). Superintendents annual report for fiscal year 20012002. Phoenix: Author.
Association for Direct Instruction. (2003). General FAQs. Retrieved May 18, 2007, from http://adihome.org/index.php?option=com_content&task=view&id=12&Itemid=31
Ballou, D., Teasley, B., & Zeidner, T. (2006). A comparison of charter schools and traditional public schools in Idaho. Nashville, TN: Vanderbilt University, National Center on School Choice.
Bifulco, R., & Ladd, H. F. (2006). The impacts of charter schools on student achievement: Evidence from North Carolina. Education Finance and Policy, 1, 5090.
Booker, K., Gilpatric, S. M., Gronberg, T. J., & Jansen, D. W. (2004). Charter school performance in Texas. Knoxville: University of Tennessee.
Buddin, R., & Zimmer, R. (2005). Student achievement in charter schools: A complex picture. Journal of Policy Analysis and Management, 24, 351371.
Bulkley, K. E. (2002, April). Balancing act: Educational management organizations and charter school autonomy. Paper presented at the American Educational Research Association, New Orleans, LA.
Center for Education Reform. (2006). Charter schools: Changing the face of American education (9th ed.). Washington, DC: Author
Chall, J. S. (2000). The academic achievement challenge: What really works in the classroom. New York: Guilford Press.
Charter School Achievement Consensus Panel. (2006). Key issues in studying charter schools and achievement: A review and suggestions for national guidelines. (NCSRP White Paper Series, No. 2).
Seattle: University of Washington, National Charter School Research Project.
Charter schools; purpose, scope, Arizona Rev. Stat. 15-181 (1994). (Amended 1998)
Charter schools; requirements, Arizona Rev. Stat. 15-183 (1994). (Amended 2005)
Comprehensive School Reform Quality Center. (2006). CSRQ Center report on education service providers. Washington, DC: American Institutes for Research.
Conn, K. (2002). For-profit school management corporations: Serving the wrong master. Journal of Law and Education, 31, 129148.
Core Knowledge Foundation. (2007). Frequently asked questions. Retrieved May 18, 2007, from http://coreknowledge.org/CK/about/FAQ/FAQ.htm#10
Dryden, M. (2004, April). The performance of Edison Schools Inc. in the Dallas schools. Paper presented at the American Educational Research Association, San Diego, CA.
Edison Schools Inc. (2006). Edison Schools eighth annual report on school performance 20042005. Retrieved December 30, 2006, from http://www.edisonschools.com
Garn, G. (1999). Solving the implementation problem: The case of Arizona charter schools [Electronic version]. Education Policy Analysis Archives, 7(26). Retrieved May 17, 2007, from http://epaa.asu.edu/epaa/v7n26.html
Gill, B. P., Hamilton, L., Lockwood, J. R., Marsh, J. A., Zimmer, R., Hill, D., et al. (2005). Inspiration, perspiration and time: Operations and achievement in Edison Schools. Santa Monica, CA: RAND.
Gomez, J. J., & Shay, S. A. (2000). Evaluation of the Edison Project schools: Third interim report: 19981999 school year. Miami, FL: Miami-Dade County Public Schools.
Gresham, A., Hess, F., Maranto, R., & Milliman, R. (2000). Desert bloom: Arizonas free market in education. Phi Delta Kappan, 18, 751757.
Gronberg, T. J., & Jansen, D. W. (2001). Navigating newly chartered waters: An analysis of Texas charter school performance. Austin: Texas Public Policy Foundation.
Hanushek, E. A., Kain, J. F., & Rivkin, S. (2002). The impact of charter schools on academic achievement Unpublished manuscript, Stanford University, Palo Alto, CA.
Hanushek, E. A., Kain, J. F., Rivkin, S., & Branch, G. (2005). Charter school quality and parental decision making with school choice. New York: National Bureau of Economic Research.
Harris, D. C. (2006). Lowering the bar or moving the target: A wage decomposition of Michigans charter and traditional public school teachers. Educational Administration Quarterly, 42, 424460.
Hassel, B. C. (1999). The charter school challenge. Washington, DC: Brookings Institution.
Henig, J. R., Holyoke, T. T., Brown, H., & Lacireno-Paquet, N. (2005). The influence of founder type on charter school structures and operations. American Journal of Education, 111, 487522.
Hentschke, G., Oschman, S., & Snell, L. (2002). Education management organizations: Growing a for-
profit education industry with choice, competition and innovation. Los Angeles: Reason Public Policy Institute.
Hess, F., & Leal, D. L. (2003). An evaluation of student performance in National Heritage Academies charter schools: 20002003. Washington, DC: American Enterprise Institute.
Hess, F., Maranto, R., Milliman, R., & Grammatico Ferraiolo, K. (2002). In the storms eye: How race, experience and exposure shape Arizona teachers attitudes toward school choice. Teachers College Record, 104, 15681590.
Horn, J., & Miron, G. (2000). An evaluation of the Michigan charter school initiative: Performance, accountability and impact. Kalamazoo: Western Michigan University, The Evaluation Center.
Lacireno-Paquet, N. (2004). Do EMO-operated charter schools serve disadvantaged students? The influence of state policies [Electronic version]. Education Policy Analysis Archives, 12(26). Retrieved August 15, 2004, from http://epaa.asu.edu/epaa/v12n26/
Lake, R. J., & Hill, P. T. (Eds.). (2005). Hopes, fears, and reality: A balanced look at American charter schools in 2005 (Annual Report). Seattle: University of Washington, National Charter School Research Project.
Leona Group. (2006). Leona Group LLC: A new kind of public school. Retrieved March 3, 2007, from http://www.leonagroup.com/index1.html
Levin, H. (2002). Potential of for-profit schools for educational reform. New York: Columbia University, Teachers College.
Levin, H. (2006). Why is this so difficult? In F. M. Hess (Ed.), Educational entrepreneurship (pp. 165182). Cambridge, MA: Harvard Educational Press.
Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29(2), 416.
Loschert, K., ONeil, J., & Winans, D. (2004, September). Cash cow [Electronic version]. NEA Today. Retrieved September 15, 2007, from http://www.nea.org/neatoday/0409/coverstory.html
Loveless, T. (2003). Charter schools, achievement, accountability and the role of expertise. In 2003 Brown Center report on American education: How well are American students learning? (pp. 2736).
Washington, DC: Brookings Institution.
Maranto, R., & Maranto, A. G. (2006). Markets, bureaucracies, and clans: The role of organizational culture. In F. M. Hess (Ed.), Educational entrepreneurship (pp. 145164). Cambridge, MA: Harvard Educational Press.
Matczynski, T. J., Rogus, J. F., Lasley, T. J., II, & Joseph, E. A. (2000). Culturally relevant instruction: Using traditional and progressive strategies in urban schools. The Educational Forum, 64, 350357.
McEwan, P. J. (2000). The potential impact of large-scale voucher programs. Review of Educational Research, 70, 10349.
Medler, A. (2004). Charter schools: Complementing or competing with public education? In E. Rofes & L.
M. Stulberg (Eds.), The emancipatory promise of charter schools (pp. 189218). Albany: State University of New York Press.
Minneapolis Public Schools. (2000). Edison project school information report. Minneapolis, MN: Author.
Miron, G. (2007). Education management organizations. Manuscript in preparation.
Miron, G., & Applegate, B. (2000). An evaluation of student achievement in Edison Schools opened in 1995 and 1996. Kalamazoo: Western Michigan University, The Evaluation Center.
Miron, G., & Nelson, C. (2002). Whats public about charter schools? Lessons learned about choice and accountability. Thousand Oaks, CA: Corwin Press.
Molnar, A. (2001). Calculating the benefits and costs of for-profit public education [Electronic version]. Education Policy Analysis Archives, 9(15). Retrieved April 30, 2001, from http://epaa.asu.edu/epaa/v9n15.html
Molnar, A., Garcia, D. R., Bartlett, M., & ONeill, A. (2006). Profiles of for-profit education management organizations, eighth Annual Report, 20052006. Tempe: Arizona State University, Commercialism in Education Research Unit.
Molnar, A., Garcia, D. R., Sullivan, C., McEvoy, B., & Joanou, J. (2005). Profiles of for-profit education management organizations, seventh Annual Report 20042005. Tempe: Arizona State University, Commercialism in Education Research Unit.
Molnar, A., Wilson, G., & Allen, D. (2004). Profiles of for-profit education management companies, sixth annual report 20032004. Tempe: Arizona State University, Commercialism in Education Research Unit.
National Heritage Academies. (2007). Measurable results. Retrieved March 3, 2007, from http://www.nationalheritageacademies.com/about-us/measurable-results/
Nelson, C., & Hollenbeck, K. M. (2001). Does charter school attendance improve test scores? Comments and reactions on the Arizona achievement study. Kalamazoo, MI: W. E. Upjohn Institute.
Nelson, F. H., Drown, R., Muir, E., & Van Meter, N. (2001). Public money and the privatization of K12 education. In S. Chaikind & W. Fowler (Eds.), Education finance in the new millennium: 2001 Yearbook of the American Education Finance Association (pp. 173192). Larchmont, NY: Eye on Education.
Nelson, F. H., & Van Meter, N. (2003). Student achievement in schools managed by Mosaica, Inc. Washington, DC: American Federation of Teachers.
Nichols, S., Glass, G. V., & Berliner, D. (2006). High-stakes testing and student achievement: Does accountability pressure increase student learning? [Electronic version]. Education Policy Analysis
Archives, 14. Retrieved March 20, 2007, from http://epaa.asu.edu/epaa/v14n1/
Sass, T. R. (2006). Charter schools and student achievement in Florida. Education Finance and Policy, 1, 91122.
Seltzer, M. H., Frank, K. A., & Bryk, A. S. (1994). The metric matters: The sensitivity of conclusions about growth in student achievement to choice of metric. Educational Evaluation and Policy Analysis, 16, 4149.
Shay, S. A. (2000). A longitudinal study of achievement outcomes in a privatized public school: A growth curve analysis. Unpublished doctoral dissertation, University of Miami, Coral Gables, FL.
Solmon, L. C., Paark, K., & Garcia, D. R. (2001). Does charter school attendance improve test scores? The Arizona results. Phoenix, AZ: Goldwater Institute.
Harcourt Assessment. (1997). Stanford achievement test series: Spring norms book (9th ed.). San Antonio, TX: Author.
Success for All Foundation. (2005). Press room: Training. Retrieved May 18, 2007, from http://www.successforall.net/press/press_training.htm
U.S. Government Accountability Office. (2002). Public schools: Insufficient research to determine effectiveness of selected private education companies (GAO-03-11). Washington, DC: Author.
U.S. Government Accountability Office. (2003). Public schools: Comparison of achievement results for students attending privately managed and traditional schools in six cities. Washington, DC: Author.
Wolfram, G. (2002). Making the (better) grade: A detailed statistical analysis of the effect of National Heritage Academies on student MEAP scores. Hillsdale, MI: Hillsdale Policy Group.
Zimmer, R., & Buddin, R. (2005). Getting inside the black box: Examining how the operation of charter schools affects performance (Working Paper). Santa Monica, CA: RAND.
Table A1. Regression Coefficients, Achievement Model, Same Sector All 3 Years
Table A2. Regression Coefficients, Achievement Model, Same School All 3 Years
Table A3. Student Demographic and Academic Characteristics, by Sector of Attendance, For Those Who Stayed in the Same School All 3 Years, 2001
a Number of students. All other figures are percentages of students in that category.
b Accommodations and special education are not mutually exclusive. A student can receive accommodations on the test without being classified as special education.