The Unbalanced Equation
by Harvey Averch - 1973
Review of Inequality: A Reassessment of the Effect of Family and Schooling in America (1972).
Harvey Averch is division director, Social Systems and Human Resources/ Research Applications, National Science Foundation. The views expressed in this paper are the author's own and do not represent an official opinion of the National Science Foundation.
Inequality: A Reassessment of the Effect of Family and Schooling in America
Christopher Jencks, Marshall Smith, Henry Acland, Mary Jo Bane, David Cohen, Herbert Gintis, Barbara Heyns, and Stephan Michelson. New York: Basic Books, 1972. $10.00. 399pp.
Inequality has been widely reviewed. From both the technical and policy perspectives, it has received extensive coverage. Yet very little to date has been written on this controversial study from the perspective of the applied research funder or the decision-maker who has scarce resources to buy policy research. This review, therefore, will address the cost-effectiveness of Inequality.
According to the preface, its eight authors devoted three years to producing Inequality. The Carnegie Corporation sponsored some of the work through institutional grants to the Harvard Center for Educational Policy Research. Grants and contracts to the Center from the Department of Health, Education, and Welfare; Office of Economic Opportunity; Urban Institute; and Massachusetts State Department of Education also helped support the work. The authors do not tell us the "total systems cost" of Inequality. A reasonable estimate would be $500,000.1
Stated bluntly, we need to ask whether the policy information and analysis in Inequality is commensurate with the research investment. To answer this question, we need the answers to a series of collateral questions about the specification of the issues to be analyzed and the research strategies chosen. Unfortunately, Inequality does not contain the documentation required to make a Complete judgment about either its analytical merits or its policy utility.
The authors do not tell us whether they consciously considered alternative specifications of the issues and alternative research strategies or designs for inquiry. We do not know, for example, why the secondary analyses of existing studies and data were chosen as opposed to original inquiry, nor do we know the rules for the synthesis and integration of the secondary analyses. Further, we do not have the terms of reference laid out by the sponsors of the work so that we can test how well the analysis matches what was expected.
The policy questions of Inequality have to be inferred from the structure of the book or from reading other materials, such as Jencks's rebuttal statement in the Harvard Educational Review.2 The policy question appears to be whether, for society as a whole, reducing variation in access to and treatment within major social institutions reduces variation in income. The educational system receives much more weight in the analysis and much more research effort than, say, the health care system. I infer from this that special focus on the educational system was selected because the authors believe that major errors in educational policy were induced by erroneous policy assumptions. Someonesociety, unidentified policy-makersbelieved that increasing educational opportunity and access would reduce income inequality. This assumption, the authors imply, induced erroneous resource allocations and educational policy. Argumentation and evidence for this belief is not strongly presented, nor do the authors demonstrate that operating educational programs came into being because the bureaucrats and policy-makers believed that income inequality would be reduced as a result of the programs. In sum, Inequality does not precisely specify its questions nor address in a structured way policy-makers at different levels within the educational system.
This condition leads to my second question: For an expenditure of approximately $500,000 and three years, what is the incremental gain in actionable knowledge provided by Inequality? Since the Jencks criterion for evaluating a social systems' effectiveness is explained variance in income inequality, and no social system by itself or in combination explains much of the variance in income inequality, the policy that the authors suggest is that income inequality be treated directly through income transfers and taxes. This preferred policy is arrived at by a process of eliminating other social institutions as viable alternatives,3 not by evaluating the effects of the preferred alternative.
However, the authors also argue that resources should not be diverted from the ineffective social systems because those systems' inner values are worth preserving. Thus the authors' policy message is ambiguous. They sidestep the hard choices that appear to flow from their analysis, for they implicitly deny that society has any resource constraints. Thus, they do not have to accept the implications of their own analyses. For example, they argue that the educational system should continue to receive resources in order to improve the "quality of life" of those in the schools. But other, perhaps more effective, ways of improving the quality of life of youth, teachers, or administrators are not investigated.
One could at least argue for direct-income transfers to children or parents to improve the quality of life.
In any case, Inequality does not suggest programs along with their cost-benefits for policy-makers. How to get from here to their preferred alternative is left completely unexplored. The authors could have used their research dollars to assert at the beginning that income inequality is too great and could have spent three years carefully evaluating the benefit and costs and feasibility of alternative income redistribution programs.4
This raises a third query: Is the research design and strategy of Inequality carefully linked to the problem and to the conclusions? Stated another way, given the limitations of the original research studies on schools, is very intensive secondary analysis a cost-effective procedure; and is a decision-maker's confidence for action or inaction enhanced by pooling the results of studies, each of whose questions and design differs? The answer depends on the design for pooling the results and on whether the different results provide some cross-check on each other. Most importantly, it depends on the policy-maker knowing the rules for study selection and on being able to reproduce or check the synthesized information with a minimum of time and effort. Inequality is ambiguous in stating its rules of the game. It does not clearly explain how estimates are made or the logic of its estimates. The language used is "our best estimate." But it is difficult to know whether adjustments were made to an estimate from secondary analysis, whether the authors proceeded on intuition, or from the gestalt (perhaps desperation) induced by three years of immersion in the data.
Another test of Inequality concerns its design for utilizationwhether a policymaker's behavior is changed as a result of the research results. Utilization is a "marginal" concept in that there are many other issues besides research results involved in taking action. If behavioral scientists believe their work is relevant for policy, then it must be designed and presented so that it can be used. Inequality is clearly written as an academic treatise with other academics in mind as readers. We are not told what reports the sponsors of the work received, but the book cannot be used by policymakers without "translation." There is no "executive summary" explaining the findings, their action implications, principal methods, uncertainties in inference, and most importantly, a statement by the authors on how much confidence they have in their own analysis. (One crude test might be how much and at what odds they would wish to bet on the correctness for action of their analysis if they were required to return research dollars to their sponsors when wrong.)
The authors also give no indication of future efforts to get their results utilized or whether they are at all concerned about use. It is unclear whether any of the research costs were to be allocated to utilization. Of course, it can be argued that the authors' utilization plan is an indirect one, that by appealing to scholars and intellectuals, rather than to policy-makers, they will eventually change policy through altering the climate of opinion. But if this is the strategy, then it should be made explicit. It is a weak strategy for policy research, since almost any kind of research and allocation of research dollars can be justified on those grounds.5
Since Inequality recommends political control over United States economic institutions, it is important to establish whether that conclusion can really be derived from analysis or from the authors' beliefs. If from analysis, it is important to know the critical arguments and sensitivities that would cause the conclusion to be overturned. Surely, one can urge "socialism" to reduce inequality without spending three years in regression or path analysis, but if one does, then one should clearly specify how the conclusion flows from the analysis and why the alternative being advocated will turn out to be better than the current system.
In evaluating a piece of research tike Inequality, it is necessary to examine not only the work itself but the system of evaluation and review. As noted, Inequality has been widely reviewed, but not with a systematic design, for it is not customary among sponsors of research to make resources available for evaluation and review. While the standard academic review and rebuttal are adequate for basic research, the controlling questions on policy research mandate a different design and a different kind of evaluation. As a piece of policy research, Inequality would be far more effective if it were published together with its various critiques. In addition, a short paper pointing out areas of agreement and disagreement, and agreements to disagree, would be valuable. Otherwise, contrary to the authors' hopes, the only response to Inequality will be more research proposals to refute Inequality, which proposals, if funded, will generate other requests for research dollars.6 While researchers will benefit from this behavior, policy-makers will not.