The Proficiency Trap: New York City’s Achievement Gap Revisited

On July 17, 2008, New York City Mayor Michael Bloomberg and Schools Chancellor Joel Klein appeared before the U.S. House Committee on Education and Labor and argued that New York’s version of accountability had substantially closed the achievement gap. Klein said:

Since we started this work in 2002....our African-American and Latino students have gained on their white and Asian peers. In fourth-grade math, for example, the gap separating our African-American and white students has narrowed by more than 16 points. In eighth-grade math, African-American students have closed the gap with white students by almost 5 points. In fourth-grade reading, the gap between African-American and white students has narrowed by more than 6 points. In eighth-grade reading, the gap has closed by about 4 points. (Klein, 2008, p. 3)

Bloomberg echoed that claim, arguing that, “over the past six years, we’ve done everything possible to narrow the achievement gap—and we have. In some cases, we’ve reduced it by half” (Bloomberg, 2008, p. 2).

Over a brief interval of five years, was it possible that New York City schools had made more progress in closing the achievement gap than the rest of the country had in the previous thirty? In reality, the average African American and Hispanic students in New York City are as far behind their white and Asian peers as they were when Mayor Bloomberg assumed control of the New York City schools. Still, Bloomberg and Klein’s ability to claim that the achievement gap had narrowed offers important lessons for education policymakers about the dangers of measuring inequality between groups by relying on passing rates.

Bloomberg and Klein’s assertion was based on differences in the proficiency rate, or the percentages of students meeting the cut score that New York state defines as proficient. But proficiency is a misleading and inaccurate way to measure achievement gaps, though political actors often prefer to employ this metric because it paints a more positive picture of progress than truly exists. Primarily, the problem is that we cannot differentiate between students who just made it over the proficiency bar and those who scored well above it. Proficiency rates can increase substantially by moving a small number of kids up a few points—just enough to clear the cut score—or by shifting the cut score itself down. But African American and Hispanic students may still lag far behind their white and Asian peers even as their proficiency rates increase.

The deceptive nature of proficiency rates becomes obvious when looking at gaps in New York City student scale scores—a continuous measure of student progress—on the state tests between 2003 and 2008. Though Chancellor Klein claimed a reduction in the achievement gap for all grade levels and subjects, Table 1 demonstrates that for 4th grade math and reading tests, as well as 8th grade math tests, the achievement gap has either grown or declined by a meager amount. Where we observe declines, they are less than 0.05 standard deviations. (See Table 1.) There is a modest decline in gaps for 8th graders in language arts, with the greatest decline being the African American-Asian gap, from 0.66 standard deviations in 2003 to 0.52 standard deviations in 2008. Even in this case, this reduction falls considerably short of Bloomberg’s claim that the achievement gap had been halved.

Table 1: Gaps in mean scale scores, New York state tests in English language arts and math, 2003 and 2008 (standard deviation units)

Source: New York City Department of Education (2008). In each case, the standard deviation used as the denominator was the group with the higher score. Using the mean of the two groups’ standard deviation changes the results little.

In addition, New York City students’ scale scores from NAEP’s Trial Urban District Assessment (TUDA) show no progress in closing achievement gaps. In 2003, 2005, and 2007—the time period that can be plausibly attributed to the Bloomberg administration’s policies - a sample of fourth- and eighth-grade students in New York City participated in NAEP’s urban testing program in both reading and math. The National Center for Educational Statistics’ analyses demonstrated that there were no statistically significant changes in African American-white or Hispanic-white gaps between 2003 and 2007. Based on our own calculations, we find that African American-Asian and Hispanic-Asian gaps in 8th grade reading, and the Hispanic-Asian gap in math, have grown substantially and these differences are statistically significant.

Despite Bloomberg and Klein’s claims that the achievement gap has narrowed, the basic facts about New York City’s achievement gap are sobering. In 2007, the average African American 4th grader in New York City performed at the 20th percentile of the white test score distribution on the NAEP math test, and the 14th percentile of the Asian test score distribution.¹ Put differently, 80% of white 4th graders performed above the average African American score, and 86% of Asian 4th graders did. These are hardly figures to gloat about, especially as they have barely budged over the current administration’s tenure.

Yet the political uses of these test score data are also not surprising. As Starr (1987) notes,

An average is not just a number; it often becomes a standard…. Many regularly reported social and economic indicators have instantly recognizable normative content. The numbers do not provide strictly factual information. Since the frameworks of normative judgment are so widely shared, the numbers are tantamount to a verdict. (p. 54)

As the New York legislature debates the renewal of mayoral control of the city school system, we should expect intense and selective uses of data to fuel arguments on each side. The selective use of data in New York City has begun to come under some scrutiny (e.g., Green, 2008), and whether it will continue to do so in the midst of intense lobbying (Medina & Gootman, 2008) is an open question. Unfortunately, the primary losers in this political game are New York City’s African American and Hispanic children, who will ultimately face the consequences of this gap when they compete with their peers in the classroom and in the workplace.

Note

1. We assume a normal distribution of scores.

References

Bloomberg, M. R. (2008, July 17). Testimony of Michael R. Bloomberg, Mayor, the City of New York, before the House Committee on Education and Labor on mayor and superintendent partnerships in education: Closing the achievement gap [online]. Retrieved August 26, 2008, from http://edlabor.house.gov/testimony/2008-07-17-MichaelBloomberg.pdf.

Gootman, E. (2003, October 22). Math scores rise sharply across state. New York Times [online]. Retrieved August 30, 2008, from http://query.nytimes.com/gst/fullpage.html?res=9B05E1DE1631F931A15753C1A9659C8B63&sec=&spon=&partner=permalink&exprod=permalink.

Green, E. (2008, August 5). “Achievement Gap” in city schools is scrutinized: Slight gains in English are reported. New York Sun [online]. Retrieved August 26, 2008, from http://www.nysun.com/new-york/achievement-gap-in-city-schools-is-scrutinized/83215/.

Klein, J. I. (2008, July 17). Testimony of Joel I. Klein, Chancellor, the City of New York, before the House Committee on Education and Labor on mayor and superintendent partnerships in education: Closing the achievement gap [online]. Retrieved August 26, 2008, from http://edlabor.house.gov/testimony/2008-07-17-JoelKlein.pdf.

Lutkus, A., Grigg, W., & Dion, G. (2007). The Nation's Report Card: Trial Urban District Assessment Mathematics 2007 (NCES 2008-452). Washington, DC: National Center for Education Statistics, Institute of Education Sciences.

Lutkus, A., Grigg, W., & Donahue, P. (2007). The Nation’s Report Card: Trial Urban District Assessment Reading 2007 (NCES 2008–455). Washington, DC: National Center for Education Statistics, Institute of Education Sciences.

Medina, J., & Gootman, E. (2008, September 1). Campaign to keep schools under mayor’s thumb. New York Times [online]. Retrieved September 1, 2008, from http://www.nytimes.com/2008/09/02/nyregion/02control.html?ex=1378094400&en=23c3acf902b48367&ei=5124&partner=permalink&exprod=permalink.

New York City Department of Education. (2008). New York City City and State Test Scale Score Data 2002-2008 [spreadsheet]. New York: Author. Retrieved July 30, 2008, from http://spreadsheets.google.com/pub?key=pp3Ho_3PCkoNCPjoWbf2BMQ.

Starr, P. (1987). The sociology of official statistics. In P. Starr & W. Alonso (Eds.), The politics of numbers (pp. 7–57). New York: Russell Sage Foundation.