Did the Coleman Report Underestimate the Effect of Economic Status on Educational Outcomes?
by John L. Rury & Argun Saatcioglu - January 22, 2015
This research note considers the historical question of how well the proxy variable used in the Coleman Report to represent economic status actually predicted educational outcomes. We use U.S. Census data from 1960 to construct proxies similar to Colemanís and compare them to a variable directly measuring relative economic status, adjusted for family size. Our analysis suggests that such proxies under-estimate the economic standing of many households and that these indicators do not predict educational outcomes as effectively as more direct measures of income. We conclude with a brief discussion of implications of this finding for educational researchers today.
The 1966 Equality of Educational Opportunity (EEO) report by James Coleman has been described as a seminal moment in the history of educational research. The first large scale survey of educational achievement in the United States, initiated by the 1964 Civil Rights Act, it aimed to determine whether disparities in school resources were linked to racial differences in educational outcomes. Colemans findings were a shock at the time; relatively little variation in achievement appeared to be due to such institutional differences. (Coleman, 1969) Instead, it was family background factors that were most clearly linked to achievement. As Coleman wrote, schools are not acting as a strong stimulus independent of the child's background, or the level of the student body. (Coleman, 1966, p. 311) Even if subsequent analyses of Colemans data have revealed that schools played a bigger role in determining outcomes than he believed, the basic point that family background is the principal predictor of achievement has been described as a seminal finding in U.S. sociology of education. (Gamoran and Long, 2006. p. 1; Borman and Dowling, 2010).
Our purpose is not to challenge this essential insight. Rather, we intend to examine whether Coleman underestimated the effect of economic status, a core element in his operationalization of family background factors that turned out to be so important in the EEO report. It is important to point out that economic status, as Coleman measured it, was only one component of contemporary indicators of socioeconomic status (SES), as it did not include such key components as parental occupation and education. Coleman specified parental education as a control measure separately from economic status. The use of both of these factors contributed to the power of student background in his analysis, but we focus just on economic status in this discussion.
As discussed below, Coleman measured economic status as combination of various household possessions, urbanicity, and family size. This was his method for constructing a proxy for the economic circumstances of the students family. We replicate Colemans approach and compare it to the predictive strength of a direct and more accurate measure of family income from the 1960 census, while controlling for other background factors such as parental education, race, and gender. We believe this is useful both from a historical standpoint, and because of the methodological implications it holds for other studies of educational outcomes. In this respect, our basic questions in the discussion to follow can be framed in this way: Was Colemans estimate of the effect of economic status on educational performance lower than what it should have been? And if so, how much lower?
EVALUATING COLEMAN WITH CENSUS DATA
Colemans survey was large and comprehensive, and focused specifically on determining the effect of resource differences on educational outcomes. In addition to information on students, it included surveys of teachers and principals, providing a good deal of information about educators and institutions at the time. Altogether, it was quite expansive and was designed to determine how much difference in student achievement was due to institutional inequities. Consequently, his team devoted relatively little effort to collecting information on the economic status of the more than 600,000 students they surveyed in 1965. Instead, as noted in the first chapter of the EEO report, statistics having to do with the pupils personal socioeconomic backgrounds, level of education of their parents and certain items in their homes (such as encyclopedias, daily newspapers etc.) are based on student responses to questionnaires. (Coleman, 1966, p. 8)
In short, Coleman asked students to provide information on their economic status, and it took the form of asking if their families owned a TV set, a telephone, a record player, a refrigerator, an automobile or a vacuum cleaner. They were also asked about the type of jobs their fathers held, although this information was incomplete and was not used in the analysis. Other questions were directed at reading materials in the home, family size and structure, parental education and attitudes toward school and a number of other matters. But on the question most directly associated with the economic status of the household, this brief inventory and a few other questions (including family structure) were intended to provide a set of indirect measures to create an aggregate proxy measure of economic circumstances without having to take the costly step of distributing questionnaires to parents1 (Coleman, 1966).
Coleman was aware that the Reports measures of family background were limited. There are, of course, many other aspects of the child's background that are not measured here, he wrote, thus the variance accounted for by these variables can be interpreted as a kind of lower limit to the actual effects of background differences. (Coleman, 1966, pp. 299300) But it has been left to others to interpret just what these variables may represent. In this paper, we evaluate the use of household items such as these and other variables to represent economic status. Specifically, we use census data from the Integrated Public Use Microdata Series (IPUMS) to consider just how effectively a set of variables such as this function as a substitute for more direct measures of resources. (Ruggles, et. al. 2010)
In 1960 and 1970 the U.S. Census included questions on the ownership of certain household appliances for a 20% sample of the population, including three of the items on Colemans list: televisions, telephones and automobiles. Four other items represented functions similar to the remaining Coleman appliances: radios, air conditioners and washers and dryers. If anything, these household conveniences probably represented an even wider array of income and status categories than Colemans more commonplace items. We used these data to estimate just how such items compare with a more accurate income indicator, which incorporates direct measures of income. While any measure of income is prone to both measurement and estimation errors, we wonder whether an economic status measure based upon indicators beyond household possessions would help estimate a more accurate effect on educational performance.
Using IPUMS, we have constructed a national sample of 17 year-olds in 1960 (n=3,832). In doing this we focus on the oldest group in the Coleman sample: high school seniors. The date is proximate to Colemans survey, and prior to the many policy changes that dramatically reduced the numbers of Americans living in poverty in the decade to follow and rapidly rising attainment levels. The principal objective of our analysis is to contrast the effect size of Colemans proxy measure with a more direct measure of economic status available in IPUMS.
MEASURING FAMILY ECONOMIC STATUS
We begin by describing the IPUMS sample and some of the key indicators that it offers for an analysis of this type.2 Basic descriptive information for all measures used herein are shown in Table 1. The 1960 U.S. Census collected information on income but not on poverty or other dimensions of economic status. IPUMS samples, however, offer a helpful set of variables to construct a multifaceted and rigorous economic status measure. The most complete indicator of economic status in IPUMS is tagged as Poverty Status, representing total family income expressed as a multiple of the 1999 federal poverty rate, adjusted for family size and inflation. It is calculated as: ([family income] / [poverty threshold for that family size]) x 100, such that a family with income at fifty percent greater than its associated poverty level would be scored at 150. A family with income at fifty percent of (or below) the poverty level would have a score of 50. In this way every family is given a score between 1 and 501, with those at the top earning five times the poverty level, or greater than $115,000 for a family of four in 2012. This measure is a robust indicator of economic standing, estimated in a consistent manner and adjusted for family size. (Ruggles, et. al., 2010).
Table 1. Measures Used in the Analysis
The second economic status indicator from IPUMS involves a proxy measure similar to that constructed by Coleman. The Coleman Report relied on a number of components in this regard. The first of these was the inventory of six household items intended to represent variation in disposable income. Each item (TV, telephone, record player, refrigerator, automobile, and vacuum cleaner) was measured on a dichotomous scale (e.g., Do you have a TV set? y/n). The next component was Structural Integrity of the Home, a variable indicating whether either or both parents lived with the student, or if stepparents or other surrogates fulfilled these roles. Family size (number of siblings) and urbanism of background completed the scale. The latter variable was based on a question that provided students a range of possible answers, including living in a medium sized, a large or a very large city. Coleman aggregated standardized scores for these individual measures to develop an economic status proxy. (Coleman, 1966).
Several items in the IPUMS data can be used to construct measures that parallel those found in the Coleman Report. The seven household items available in IPUMS (Coleman had six) are one dimension of this, including telephone, TV set, radio, washer, dryer, air conditioner, and automobile (0/1). Consistent with Colemans original approach, the second component of economic status is family size, measured in IPUMS as the number of family members present in the household, analogous to Colemans number of siblings. The third component is urbanism, indicated in IPUMS by residence in the central city of a metropolitan area (0/1). The fourth component is family structure, indicated by the presence of both parents and/or parent-surrogates in the home (0/1). Following Coleman, we aggregated standardized scores for these individual measures to develop an economic status proxy.3
We also created an economic status measure that is a combination of total family income and the Coleman economic status composite. As we will demonstrate in discussing our findings, this combined measure helps examine if the Coleman proxy indicator represents information that is unique or distinct from the family income measure. These latter factors are correlated, as expected, although their moderate magnitude of association (.49) suggests that the proxy captures only about a quarter of the variance in economic status represented by the income measure, or vice versa.
As seen in Table 1, we rely on three important control measures consistent with those used by Coleman in his original approach to adjust the estimate of the effect of his economic status proxy to other important demographic and background factors. The first is parental education. With more than 66 possible levels, this is a more fine-grained measure than the corresponding variable in the Coleman Report, although the IPUMS data do not provide information on family reading materials or parental educational attitudes, both included in Colemans Family Education Level. For two-parent households, we averaged the highest grade completed by the mother and the father, or the parental surrogates. For single-parent households, we used the score for the resident parent, or the parental surrogate. Our second and third control measures respectively are race and gender. Finally, Table 1 also shows basic descriptive statistics regarding our outcome measuregrade 11 or higher attainmentwhich we discuss below.
THE EFFECT OF ECONOMIC STATUS ON ATTAINMENT
Colemans study was among the first to examine educational outcomes, and that accounted for much of its impact. The dependent variable was achievement, measured by a battery of standardized tests. Unfortunately we cannot link IPUMS data to achievement scores at the individual level, so we have opted to use attainment as an outcome measure for the purpose of comparing the different economic status indicators we have constructed. Attainment and achievement are related in a general fashion, of course, as individuals with greater achievement scores tend to reach higher levels of attainment (Roscigno, Tomaskovic-Devey and Crowley, 2006). The link is imperfect, however, as test scores are an imprecise predictor of high school and college graduation and other attainment milestones. Still, there is evidence that the association between achievement and attainment was considerably stronger in the past, when high school and college graduation was much less commonplace (Galindo-Rueda and Vignoles, 2005). Given this, it is reasonable to assume that higher attainment in 1960 was linked to greater academic achievement.4
Accordingly, we relied on a specific attainment measure constructed with data available through IPUMS: grade 11 or higher attainment for 17-year-olds who were still enrolled or had graduated. It is a reliable measure for the likelihood of graduation because it represents a level of attainment commensurate for the cohort, with only a year to go for completion. Age 17 was at least a year beyond the legal requirement of compulsory schooling for all states in 1960. This measure is correlated with the 19 year-old graduation rate at the state level at more than 0.9, and has been used in previous studies as a proxy for graduation (Rury, Saatcioglu and Skorupski, 2010). Given that only about 66% of the 17 year-olds in our IPUMS sample were in a position to graduate within a year or had graduated, it is not unreasonable to assume that substantial differences in academic achievement distinguished them from peers who had dropped out or had been held back one or more grades in their school careers. It is necessary to use this factor because high school graduates often did not live at home, making it impossible to identify characteristics of their family households in census data. In 1960 it was still uncommon for youth who left school at this age to complete a GED or other alternative forms of secondary completion. Those youth intent on gaining a diploma, consequently, were typically required to graduate. (Rury and Saatcioglu, 2011; Saatcioglu and Rury, 2012)
As a reasonable approximation of graduation, and a broad indicator of achievement differences, this factor offers a helpful means of comparing the relationships of different measures of student economic status to a common educational outcome. To do this, we have constructed the attainment of 17 year-old youth as a dichotomous variable, with 1 representing completion of grade 11 or higher (including graduation), and 0 representing completion of a lower grade or having left school without a diploma. This permits us to employ logistic regression to examine the effects of different measures of economic status on the likelihood of attainment, and thus to compare the efficacy of Colemans construct with measures derived from the IPUMS data. To accomplish this, we fitted several models predicting the effects of economic status on the odds of grade 11 or higher attainment for 17-year-olds. The analysis also included three specific control variables, consistent with those used in the Coleman Report.
The results of the logistic regression models are shown in Table 2, where all estimates are based on standardized values, except for dichotomous race and female indicators. As seen in Model 1, net of the effect of race and gender, one standard deviation increase in the Coleman economic status proxy is related to a 5 percent (1.052) increase in the odds of grade 11 or higher attainment (p≤0.010). In Model 2, we remove the Coleman proxy and introduce our more direct measure of family income as the indicator of economic status. A standard deviation increase in this measure increases the odds of grade 11 or higher attainment by 21 percent (1.210). This effect is more than four times larger than the effect of the Coleman proxy in Model 1. The reduction in the log likelihood estimate from Model 1 for Model 2 also suggests the latter indicator fits the data better.
Table 2. Logistic Regression Models Estimating the Effect of Alternative Economic Status Measures on the Odds of Grade 11 or Higher Attainment for 17-Year-Olds in 1960
Note: N=3,837. Raw coefficient estimates are based on standardized scores. Standard errors are shown in parentheses.
In Model 3, we remove the family income measure and introduce the combined measure of economic status, which is the sum of the Coleman economic status proxy and total family income. Our objective with this combined measure is to determine whether it adds to the predictive strength of the family income measure, which would be indicated by a larger coefficient for the combined measure. As seen in Model 3, the odds ratio for the combined measure is almost exactly the same as the odds ratio for family income in Model 2, suggesting that the Coleman proxy adds nothing to the predictive strength of the direct measure of income. This suggests that the economic status information captured by the Coleman proxy is largely embedded within the family income measure, which represents information above and beyond that offered by the proxy.
Altogether, our results suggest that the Coleman economic status proxy has an effect size only about one fourth that of the direct income indicator, a finding consistent with the level of correlation between the two factors reported above. Adding the proxy to this measure does not result in improved predictive strength. However, the Coleman proxy does not mis-measure economic status. It appears simply to under-measure economic status, since it overlaps substantially with the IPUMS income measure, despite its weaker performance as a predictor of educational outcomes.
The foregoing account leaves little doubt that the Coleman Report underestimated the effect of economic status on the educational accomplishments of children in the mid-1960s. Recent analyses by Geoffrey Borman and his collaborators have demonstrated that Coleman also underestimated the contributions of school-level factors to student success, particularly in achievement. But in replicating Colemans analysis, Borman et al also used his measures of social and economic status. (Borman and Dowling, 2010; Konstantopoulos and Borman, 2011) While our analysis cannot consider direct measures of achievement as an outcome, we do point to limitations in Colemans Economic Standing factor in accounting for variation in educational attainment. Given this, it seems quite likely that the Coleman Report also underestimated the relationship of economic status to educational achievement at the time, as did the recent Borman et al re-analysis of the EEO Survey data.
As suggested earlier, we do not believe that the findings of this brief analysis fundamentally change the tenor of Colemans findings. Even with its limitations, his measure of Economic Level combined with parental education indicators was robust enough to demonstrate that socio-economic status was the principal determinant of educational success, a finding replicated in the Borman et al work. Indeed, our analysis suggests that parental education was a far more decisive factor, perhaps compensating somewhat for the weakness in his economic status measure. Thus we do not want to suggest that a dramatic reconsideration of the effect of income on school outcomes is in order. Rather than a fundamental reassessment of Coleman, we believe this discussion offers an instructive episode in the history of educational research, pointing to the need for precise measures of economic status in large-scale analyses of educational outcomes. Ultimately, the question of which measures of status are more appropriate or useful is partly an empirical matter. Coleman and his colleagues, for instance, may have imagined that television sets and automobiles were indicators of economic status while phones were far more ubiquitous, but the 1960 census revealed that more poor people owned TVs and cars than phones. As we have seen, the index of household conveniences they constructed turned out to be a relatively weak predictor of families actual economic standing.
If there is a lesson in this, it is that researchers should be quite deliberate in constructing proxy measures for social and economic status. This point was recently highlighted in a report by the National Assessment Governing Board, calling for a comprehensive standard in measuring social and economic status. (Cowin et. al., 2012) It has been echoed by Chudgar, Luschei and Fagioli (2014). Even so, Colemans approach to gauging such factors continues to be widely employed in educational research. For instance, Martin Carnoy and Richard Rothstein recently conducted an analysis of U.S students and their peers from other countries on PISA and TIMSS exams, both of which use an inventory of household items to estimate the social and economic status of children taking the tests. Carnoy and Rothstein argue for the salience of their preferred measurenumber of books in the homebut only in reference to other dimensions of the same inventory, along with mothers education. They concede, however, that counting books in homes probably does not pick up the entire social class effect on test score differences among U.S. students. (Carnoy and Rothstein, 2013, p. 88) This issue can be explored with the use of more recent datasets such as the 1988 National Educational Longitudinal Study (NELS) or the Education Longitudinal Study of 2002 (ELS), which include household inventory items, including number of books, along with information from parents on household income and family size.5 As with Colemans survey, inclusion of parental education measures offers a powerful indicator of socio-economic status and its effect on both achievement and attainment.
Asking students to report the presence or numbers of various items in their households has been a convenient way to gauge economic status since Coleman used the technique nearly fifty years ago. Our analysis has demonstrated, however, that it is a method that probably under-estimates the economic standing of many households and that such indicators do not seem to predict educational outcomes as effectively as more direct measures of income. This leaves aside, of course, the question of whether student reports of such information are accurate to begin with. While income and poverty status are typically associated with parental education, books in the home, and a number of other factors, they also appear to have a clear and robust independent relationship to outcomes such as educational attainment. As a legacy of the Coleman Report, researchers should be cognizant of these issues in future studies that consider such questions.
1. Student responses to the survey question about their fathers occupations, which provided a range of ten possible categories to choose from, were not used in constructing the Family Economic Level factor. This may have been because the data were incomplete and they provided little explanatory power over other variables in the construct. Other questions regarding students parents and their home life were utilized to construct a Family Education Level factor, which included responses regarding parental education levels, parental desires and interest in student achievement, and reading materials in the home. The education and economic level constructs represented the sum total of the Reports family background measures, which turned out to be so important in the analysis of student achievement.
2. In considering how items in the census data compared to those utilized in Coleman's analysis, it is necessary to address the question of sampling. Coleman's data were drawn from a random national stratified sample of schools, with students from selected grades surveyed separately. As a consequence, the sample did not include youth who had dropped out of school or graduated. IPUMS, on the other hand, is a nationally representative sample based on Census data, and its purview includes everyone counted by the census. Accordingly, we initially adjusted the IPUMS data to fit parameters similar to those found in the Coleman sample of high school seniors. We also performed our analyses both with IPUMS data for a sample of youth still enrolled in school and a sample representing the larger population that included dropouts.
It turns out that important differences exist between these samples. In IPUMS, for instance, we found that 21.7 percent of high school students lived in households below the poverty line in 1960, but some 41.7 percent of dropouts did so, which combined for an overall poverty rate of 25.3 percent for the national sample as a whole. Put somewhat differently, dropouts averaged 143 on the IPUMS 501 point "Poverty Status" scale, while students in school averaged 203. This suggests that Coleman's inability to consider dropouts almost certainly resulted in an underestimate of the variation in socio-economic status influencing school outcomes, a point he recognized (Coleman, 1966: 21). While dropouts represented just eighteen percent of 17 year-olds in the IPUMS data, they were some thirty percent of the sample's poor youth. Individuals in this category were excluded from Coleman's study because of its methodology.
However, in conducting analyses of the relationship of Colemans proxy indicator of economic status and the more direct measure of income, we did not find important differences between the various samples we constructed. Therefore, the results reported below reflect the full national IPUMS sample of more than 3800 individuals. Estimates based upon the different samples mentioned above, along with the approaches taken to derive such samples, are available from the authors.
3. Unlike Colemans survey, the census recorded the number of various household items, up to three in most cases. For purposes of comparison, however, we have simply counted their presence or absence of each item for purposes of comparability to the Coleman survey. For discussion of these items in the IPUMS data, see https://usa.ipums.org/usa-action/variables/group?id=h-app_mech. We are grateful to the Minnesota Population Center for making these data available to researchers.
4. As Galindo-Rueda and Vignoles (2005: 346) observed regarding achievement (ability) levels in a British cohort of students born in 1958, if a student is less able they stand a very low chance of attaining higher education, regardless of their income level. This generates a steep ability-educational attainment slope. In short, there was a direct relationship between achievement and attainment. The students in the IPUMS sample were born in 1943, fifteen years earlier than this cohort, so it is likely that the achievement-attainment relationship was broadly similar, if not even more pronounced. The ability measure employed by Galindo-Rueda and Vignoles was based in large part on achievement tests of reading and mathematics.
5. Item # 35 in the NELS 8th Grade Questionnaire asked students about an inventory of sixteen household features or items, including a typewriter, a computer, air conditioning and dedicated study space, along with 50 books or more, with a yes/no response on each. Item # 84 on the ELS 10 grade Questionnaire offers a similar but much smaller inventory of ten items, also with the 50 books or more question. Carnoy and Rothstein employ a considerably more sensitive measure of books in the home, with six categories rather than two.
Borman, Geoffrey and Maritz Dowling (2010) Schools and Inequality: A Multilevel Analysis of Colemans Equality of Educational Opportunity Data, Teachers College Record 112(5), pp. 12011246
Carnoy, Martin and Richard Rothstein (2013) What Do International Tests Really Show About U.S. Student Performance? (Washington: Economic Policy Institute)
Chudgar, Amita, Thomas F. Luschei and Loris Fagioli (2014) A Call for Consensus in the Use of Socioeconomic Status Measures in Cross-National Research using the Trends in International Mathematics and Science Study (TIMMS), Teachers College Record, http://www.tcrecord.org ID Number: 17564
Coleman, James S. (1966) Equality of Educational Opportunity (Washington, DC: U.S. Government Printing Office)
Coleman, James S. (1969) The Concept of Equality of Educational Opportunity, in Harvard Educational Review, Equal Educational Opportunity (Cambridge, MA: Harvard University Press) pp. 924.
Cowan, Charles D., Hauser, Robert, Kominski, Robert, Levin, Henry, Lucas, Samuel, Morgan, Stephen, Spencer, Margaret Beale (2012) Improving The Measurement Of Socioeconomic Status For The National Assessment Of Educational Progress: A Theoretical Foundation, National Assessment Governing Board, U.S. Department of Education.
Galindo-Rueda, Fernando and Anna Vignoles (2005) The Declining Relative Importance of Ability in Predicting Educational Attainment, The Journal of Human Resources, 40(2), 335353.
Gamoran, Adam and David. A. Long (2006). Equality of Educational Opportunity: A Forty Year Retrospective, WCER Working Paper No. 2006-9, Wisconsin Center for Educational Research, Madison, WI
Jenks, George F. (1967). The Data Model Concept in Statistical Mapping, International Yearbook of Cartography 7, pp. 186190.
Konstantopoulos, Spyros and Geoffrey Borman (2011) Family Background and School Effects on Student Achievement: A Multilevel Analysis of the Coleman Data, Teachers College Record 113(1), pp. 97132.
Roscigno, Vincent J., Donald Tomaskovic-Devey and Martha Crowley (2006) Education and the Inequalities of Place, Social Forces, 84(4), 21212145.
Ruggles, J., Steven Trent Alexander, Katie Genadek, Ronald Goeken, Matthew B. Schroeder, and Matthew Sobek. (2010) Integrated Public Use Microdata Series: Version 5.0 [Machine-readable database]. Minneapolis: University of Minnesota.
Rury, John L., Argun Saatcioglu and William Skorupski (2010) Expanding Secondary Attainment in the United States, 1940-1980: A Fixed Effects Panel Regression Model, Historical Methods, 43(3), pp. 139152.
Rury, John. L. and Argun Saatcioglu. (2011) Suburban Advantage: Opportunity Hoarding and Secondary Attainment in the Postwar Metropolitan Northeast, American Journal of Education, 118(3). 307342.
Saatciolgu, Argun. and John. L. Rury. (2012) Education and the Changing Metropolitan Organization of Inequality: A Multi-Level Analysis of Secondary Attainment in the United States, 1940-1980, Historical Methods, 45(1), 2140.