Assigning Letter Grades to Public Schools? The Danger of the Single Performance Indicator
by Jason L. Endacott & Christian Z. Goering - December 11, 2015
Sixteen states require their Departments of Education to assign a single performance indicator such as a “letter grade” to schools within those states. We take a look at the relationship between school grades and poverty in one of these states. Our analysis indicates that there is a moderate negative correlation between poverty and school performance indicators. We discuss the implications for communities and structural poverty and make a plea to reconsider the manner in which single performance indicators are determined.
In 2013, the Arkansas legislature passed two pieces of legislation requiring the state Department of Education to assign a single indicator of performance to schools in the form of an A through F letter grade. This decision was met with consternation by those who argued that the system represented an unfair oversimplification of the process of schooling. Christian Z. Goering, Associate Professor of English Education at the University of Arkansas wrote,
Grading schools based on achievement (or growth) will actually be grading them on their socioeconomic status I challenge all members of the Arkansas legislature to plop down in their cars and drive around to all of the different towns and neighborhoods in their districts, paying close attention to the size of the houses. All they have to do is count the number of garage doors they see on the houses in a particular district and then return to their offices to rank schools accordingly.
0 garage doors/carport D or F school
1 garage door D school
2 garage doors D, C, or B school
3 or more garage doors B or A school. (Goering, 2015, n.p.)
We dont have the time or the gas money to take up this challenge, but we were curious to see if there was indeed a relationship between poverty and the letter grades assigned to Arkansas schools. Researchers have found a nearly perfect correlation between parental income and scores on the SAT (0.98) & ACT (0.99) standardized tests (Orlich & Giffords, 2005). Correlations with ethnicity (0.96) were also nearly perfect. These findings do not imply causation, though other researchers have been able to predict district-level state test scores in Language Arts and Mathematics for 60 percent of school districts in New Jersey based only on percentage of single parent households, percentage of residents with at least a Bachelors degree, and percentage of economically disadvantaged children (Turnamian & Tienken, 2012). We reasoned that finding a relationship between Arkansas school grades and poverty was a likely proposition.
Thus, we ran a Pearson Correlation test to determine if there was a relationship between school scores and the percentage of students at the school who are eligible for free or reduced lunch through the National School Lunch Program (NSLP). Our analysis found that there was indeed a moderately strong negative correlation between a schools grade and the percentage of students eligible for free and reduced lunch r(1034) = -0.43, p <.001. In other words, schools with a higher percentage of students who are eligible for NSLP tend to score lower on the states report card.
Proponents of school grades might point out that this correlation could be characterized as modest, so we ran two additional statistical analyses to give the reader a better idea of how these grades look when disaggregated a bit. For the first analysis, we separated the schools into groups according to their percentage of students eligible for NSLP by assigning them a Z score (Table 1). Z scores divide a range of numbers into groups according to mean and standard deviation. Schools that are within one standard deviation above the mean have Z scores between 0 and 1. Schools that are between one and two standard deviations above the mean have Z scores between 1 and 2, and so on. The opposite is true for schools that fall below the mean. In this case, higher Z scores equate to more students eligible for NSLP.
Table 1. School Grades by Z Score
As you can see from Table 1, the wealthiest schools in Arkansas earned the highest grades and the schools with the most students eligible for NSLP received the lowest grades. The wealthiest schools scored an average of 90 points higher on the Arkansas report card than the poorest schools. The drop in scores from top to bottom is fairly consistent, and the difference between the lowest group and every other group above it is statistically significant at the p < .008 level (adjusted from p < .05 for the post hoc comparison of means).
For the second analysis we divided the groups by letter grade in order to see if there was a difference in the NSLP population between groupsessentially looking at the same question the other way around. The state assigned a letter grade system that is familiar to all of us. 90% is an A, 80% is a B, and so on.
There were 300 points possible. Table 2 illustrates our findings.
Table 2. Percentage of Students Eligible for NSLP by School Grade
Not surprisingly, schools with higher letter grades have lower percentages of students eligible for free or reduced lunch. While there were only 44 schools that were given an F, the average percentage of students eligible for NSLP in those schools was almost 83%. At the top of the report card are the schools with only 54% eligible. The differences between all of these groups is statistically significant at the p < .01 level (adjusted from p < .05 for post hoc comparison of means) with only one exception. That lone exception is the difference between the F and D schools, and is likely due to the relatively low number of failing schools, since the mean difference between D and F schools is similar to the other mean differences.
So, why does this matter? Well, it appears as though the alarms raised by the grading systems detractors might be merited. Who wants to send their kids to a C, D, or F school? Letter grades and other forms of oversimplification often detract from the positive things that schools are able to accomplish. Take Jones Elementary in Springdale, Arkansas, which has a free or reduced lunch population of approximately 98% and English Language Learner population of 80%, and received a D from the state Department of Education. When one looks beyond that mark of almost failing, it is clear that this school succeeds in many ways that a simple letter grade cannot convey. But the average citizen wouldnt know that unless they read the Education Week post written by Justin Minkel (2015), who is a Jones Elementary teacher and 2007 Arkansas Teacher of the Year. Minkels closer look reveals that Jones Elementary is one of five schools in the nation recognized by the U.S. Department of Education for academic growth, use of data, faculty collaboration, a school-based community health clinic, three pre-K classrooms, and a breakfast program that feeds every child every morning. They have introduced programs such as the Home Library Project and Parent University to help offset the punishing effects that poverty has on academic achievement. As a result, Jones Elementary has hosted many visitors hoping to learn from their success with English Language Learners and students who live in poverty. It seems almost absurd that a school with these accomplishments under its belt would be given a grade of D. The contradiction between this nearly failing grade and the schools many positive attributes and accomplishments led Minkel to conclude:
In reality, of course, policymaking in the No Child Left Behind era often penalizes children living in poverty. State legislators and data analysts may have thought that giving our school a failing grade would make us work harder (hardly possible), or take a closer look at our students who are below grade level (as if we werent already doing that). Maybe they thought they were giving families greater school choice by creating an incentive to leave Jones Elementary for a school with an A or B grade. (2015, n.p.)
This is not an issue exclusive to Arkansas as 16 different states currently have letter grade ranking systems according to the National Education Policy Center (Howe & Murray, 2015). These systems are highly problematic because research indicates that schools typically account for less than 30% of academic performance, and that emphasizing test scores with other isolated outcomes obscures the multitude of other factors that contribute to a schools performance (Howe & Murray, 2015).
Proponents of school report cards argue that the information is important for those who are deciding where to establish their families. Of course, this also has considerable and assumedly unintended consequences. Research indicates that people value school outputs (e.g., school grades) over inputs (e.g., funding) and that they are willing to pay more for a house near a school with higher standardized test scores (Downes & Zabel, 2002). In turn, these factors have an impact on property values, as schools with higher test scores and certain demographic characteristics (e.g., fewer students of color) have been found to be associated with higher home values (Clapp, Nanda, & Ross, 2008). Its pretty easy to see how this can create a self-defeating cycle for schools. Poverty is associated with outputs such as academic achievement and school grades, which are associated with demand for property and property values. Those with the means to choose where they live are less likely to live near schools with negatively perceived school outputs and demographics (as well as poverty by association), thereby leaving neighborhoods poorer and schools less likely to produce desirable outputs. This is a prime example of structural poverty, and the intersection between race and class is also a serious concern, since African American students are six times more likely to attend a high poverty school (Misra, 2015).
Supporters of the states grading system for schools argue that their statistical models demonstrate that poverty only accounts for 13% of the difference in the grades and further suggest that, While this doesnt show that the grading system is perfect, it does show that it clears a fundamental hurdle in terms of fairness (Dean, 2014, n.p.). From a positivistic perspective, the notion of controlling for poverty and determining povertys precise effect on a schools grades might seem plausible. However, from our unapologetically critical constructivist perspective, it is difficult to conceive of a situation in which one can ever isolate the effect of poverty on an entire states population of children, or even children within a single school. Children who live in poverty are more likely to suffer from food insecurity; be in poor health; have a developmental delay or learning disability; experience an emotional or behavioral problem; and be a victim of child abuse, neglect or a violent crime (Brooks-Gunn & Duncan, 1997). Childhood poverty cannot be universally described and it certainly is not universally experienced (Gorski, 2013). We argue that it is impossible to control for the effects of poverty when poverty affects every aspect of a poor childs life.
Given the problematic nature of assigning a single indicator of performance to the complex societal organism that we call school, we stand in opposition of assigning letter grades to schools. When we reflect on the excellence of schools like Jones Elementary, we are stymied by the proposition of quantifying the accomplishments of students, teachers, and administrators in a manner that would accurately reflect the conditions of learning and teaching. However, since we live in the neoliberal age of accountability, we recognize that our pleas are likely to fall on deaf ears. If states are to insist on assigning performance indicators to schools, we would implore them to consider the conclusions reached by the National Education Policy Center who write that, The indicators of school quality must be determined through authentic conversation, reflecting the voices and experiences of all members of our democratic societynot just the narrow vision of policymakers (Howe & Murray, 2015, p. 13). We must do better because the consequences of the path we are currently on are quite worrisome.
Brooks-Gunn, J., & Duncan, G. J. (1997). The effects of poverty on children. The Future of Children, 7(2), 5571.
Clapp, J. M., Nanda, A., & Ross, S. L. (2008). Which school attributes matter? The influence of school district performance and demographic composition on property values. Journal of Urban Economics, 63(2), 451466.
Dean, J. (2014, September 17). Guest blog post: AF school letter grades. Retrieved from http://officeforedpolicy.com/2014/09/17/guest-blog-post-a-f-school-letter-grades/
Downes, T. A., & Zabel, J. E. (2002). The impact of school characteristics on house prices: Chicago 19871991. Journal of Urban Economics, 52(1), 125.
Goering, C. Z. (2014, September 9). An "F" for Arkansas plan to grade schools. EduSanity. Retrieved from http://www.edusanity.com/2014/09/09/an-f-for-arkansas-plan-to-grade-schools/
Gorski, P. C. (2013). Reaching and teaching students in poverty: Strategies for erasing the opportunity gap. New York, NY: Teachers College Press.
Howe, K. R., & Murray, K. (2015). Why school report cards merit a failing grade. Boulder, CO: National Education Policy Center. Retrieved from http://nepc. colorado. edu/publication/why-school-report-cards-fail
Minkel, J. (2015, April 22). Giving excellence a 'D': When school-accountability grades fail. Retrieved from http://www.edweek.org/tm/articles/2015/04/21/ giving-excellence-a-d-when-school-accountability-grades.html
Misra, T. (2015, May 14). The stark inequality of U.S. public schools, mapped. Retrieved from http://www.citylab.com/housing/2015/05/the-stark-inequality-of-us-public-schools-mapped/393095/
Orlich, D. C. & Gifford, G. (2005). The relationship of poverty to test scores. Leadership Information, 4(3), 3438.
Turnamian, P. G. & Tienken, C. H. (2012). Use of community wealth demographics to predict statewide test results in grade 3. Paper presented at National Council of Professors of Educational Administration Conference, Kansas City, MO.