Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13
Topics
Discussion
Announcements
 

Research Quality: Quantitative or Qualitative?


by Stephen P. Heyneman - January 31, 2014

This note will report on distortions from both quantitative and qualitative research. In the end I will argue that we should throw the terms out altogether and, instead of teaching graduate students in a class labeled ‘qualitative research’, that we should teach graduate students in a class simply titled ‘research methods’.

BACKGROUND


Everyone has their own definition of qualitative research. Merriam (2009) says that the term ‘defies a simple definition’ (p. 13) because of the multiple philosophical, disciplinary, historical characteristics that underpin it. Some prefer to use the term ‘naturalistic’ or ‘interpretative’ instead of qualitative.  Some believe that the term functions so well because it is so vague that it can incorporate a plethora of techniques and orientations. These might include positivist, post-positive, interpretative, critical, postmodern and post-structural perspectives. Some suggest that the term qualitative reflects an interest in ‘understanding’ phenomena rather than (just) predicting, controlling or generalizing about it. But to researchers using empirical methods, this characterization would be patronizing. They might ask, who says that those who employ empirical techniques are not interested in understanding phenomena?  


Prior to the use of computers, social research was limited to interventions and controlled experiments. Pioneering leaders in the social sciences—Freud, Merton, and Durkheim—drew their theories primarily on the basis of observations. Analyzing survey data had been cumbersome in the 1960s and early 1970s. I remember when SPSS arrived at the University of Chicago. It was a barely finished draft manual developed by Norman Nye, a graduate student at Stanford. Yet it revolutionized the ease by which researchers, including graduate students, could analyze data and make generalizations. Essentially, it opened social research to the masses.


With ease of access, the analysis of large-scale data sets became the dominant technique in the 1980’s. Some parts of the research community had hostile reactions to this.  Observational research had been trumped in terms of more mechanical techniques. But the rise in popularity led to over-precise conclusions which can seem naïve and counter-productive in terms of policy. Reports sometimes were sloppy about explaining the assumptions of cross-sectional data, for instance, the fact that measures of association were not ‘causes’ of changes in a dependent variable.   


DISTORTIONS


Some hold that a quantitative researcher will remain objectively separated from the subject matter. They might suggest that empirical tests are ‘truly neutral’.  I think otherwise.


Take James S. Colman for instance. Throughout the 1980s, he believed that justice could be driven by scientific results. He was so convinced about the moral justice of school integration and the science supporting it, that he tended to treat ‘white flight’ away from urban school districts as a reflection of resistance to racial integration. He and I once found ourselves sharing a cab in Washington DC. “You are wrong Jim,” I said, “you know that in many countries minority children even poor minority children do not perform worse on average than others.” “Yes,” he said, “I have been reading your work on that, but why not in the United States?”


“It is true in the United States,” I said.  Those who are poor and are highly motivated, do as well as others. I reminded him of the work of John Ogbu (1991) and his own work on social capital (1988). I asked him to consider the possibility that those who flee integration are motivated for the right reasons, not the wrong reasons; that they included not only whites but minorities as well. Jim, they aren’t fleeing integration, they are fleeing something else—the likelihood that their children will experience school violence and classroom indiscipline. No responsible parent wants their child in an environment that teaches them things which are harmful. “It isn’t poverty which drives scores of American students down,” I said, “or race, or even minority status, but rather an impoverished spirit. It is the general lack of a desire to learn and this, in turn, is affected by public policy. What differentiates American from other children in the world is American public policy which assumes that curriculum has to entertain, and that students require many freedom but few obligations.”


By this time we were downtown. I remember him looking at me with that thoughtful expression. He was quiet for a long time. “Maybe,” he said. A year or so after that cab ride, I received a draft article in which he cited an essay I had written. On the draft he had written the word “Thanks” next to the citation. The point he was making in that article had to do with American adolescence. His idea of a well-balanced adolescence, and the point made that day in the cab had to do with the balance between privilege and obligations and the need of American adolescents of all races and all language groups to adhere to a common standard of behavior and performance in schools.1


James Coleman was among the ultimate in empirical researchers, yet he was deeply influenced by his emotions and as such he may have missed the true cause of what he was investigating. Neumann (2006) argues that emotion is a concomitant part of empirical research. Here is an illustration. Take today’s research on school choice, a public policy issue of major consequence. It draws the best and the brightest of education researchers and it attracts research financing from multiple public and private sources. Scholars who investigate school choice typically employ the most modern of statistical methods, including causal inference, randomized trials and hierarchical linear modeling. But the results of their complex investigations are predicable, not on the basis of the data but on the basis of the investigator’s orientation. For example, we investigated 21 school choice publications by Paul Peterson.


[39_17400.htm_g/00001.jpg]

All 21 concluded that school choice had a positive outcome. We found 19 articles by Caroline Hoxby, all 19 had positive outcomes. On the other hand we found 14 by Martin Carnoy and 6 by Rothstein; none concluded that school choice had favorable outcomes. The fact is that there is greater predictability of  the outcomes of school choice research simply by knowing the name of the researcher than on the basis of the data or methods.


Articles on school choice confuse choice with competition. Competition is a construct of social scientists; choice is a policy acted on by families. Families are in favor of choice but often could care less about competition. Even researchers who are against school choice are likely to favor a choice of particular classroom experience for their own children. By ignoring the preferences of families, all articles on school choice bypass the essence of the education policy dilemma: the fact that there is a problem of failing schools and parents often want any policy which gives them the opportunity to avoid having their children harmed by the schools they attend.


The debate on school choice has become polarized and predictable. The effect of this polarization is counter-productive for education and for education research. The public cannot trust the empirical school choice findings because they cannot trust that those findings are based on a neutral consideration of the evidence. Nor is the public assured that a few weeks later new findings using new methods and new data sources won’t reach the opposite conclusion. Because there is a low level of trust in the findings the reputation of education research declines.


In addition, the school choice research does not address the real problem. The real problem is that there are large numbers of failing schools and failing school districts. To be sure vouchers are a means to bypass this problem. Some articles conclude that choice doesn’t work. But simply saying choice doesn’t work and that we should better understand the teaching learning process (Ravitch, 2010) is not adequate and hence, not constructive.


Another example of the bias with empirical research is the fascination with national achievement scores and the tendency to blame American schools for their low international rankings in math and science. The problem with this empirical research is that it is parochial. It assumes that ‘school effects’ happen in school. In Asia ‘school’ is seven days/week. It is an intensive marathon for adolescents who attend private tutoring classes after their regular school day has concluded.  To test the effect of this ‘non-school school-effect’, we added up the studying time spent by students in Korea and the United States.


[39_17400.htm_g/00002.jpg]


We found that Korean students spend 64% more time studying than do American students. To be sure, the average score on a math test is higher for Korean students. But contrary to expectations, we found that American pupils were about 30 % more efficient in achieving each point on the PISA test than Korean students. (Heyneman, 2013). This approach was stimulated not by the search for greater empirical precision but by the results of historical and anthropological work which emphasized the importance of Korean cram schools (Tucker, 2011). All we did was quantify what anthropologists have known for years. What this suggests is that the divisions between quantitative and qualitative research techniques are artificial and that maintaining them in different categories as policy may be counterproductive.


IMPLICATIONS AND RECOMMENDATIONS


Insight is not determined by research technique. I have seen biased interviews and biased large scale surveys. I have seen randomized trials provide an elegant answer as to causality, but on a question of trivial importance. I have seen claims of sophistication in education research, but little change in the nature of the problem, that of failing schools and failing school systems. I have seen research findings used as weaponry in battles over social ideology, whether the problem of failing schools is solved through larger state financial allocations and social safety nets; or whether the problem of failing schools and systems is solved through demand side performance incentives and increased competition. In the end this research is not about schools or school children but about social ideology. And it has gotten the education community nowhere in terms of progress.


The implication of these experiences might suggest that we settle something, and that is the distinction we make between qualitative and quantitative research techniques. We sometimes treat them as if they were different categories of investigation and that they may lead to different types of insight. The distinction may have been justifiable decades ago, but today it has no discernible function and can lead to new problems. To new scholars it may suggest that one category is more culturally sensitive; that one category is more scientific; that one category can explain why things occur; that one category may insure that the findings are generalizible. Continuing to make distinctions between techniques may suggest to students that they need to choose instead of being responsible for both.


In my view we should submit ideas to tests of many kinds, including in depth discussions with respondents as well as surveys representative of larger populations. And we should always be on the lookout for the counter-factual; we should always watch for inconsistencies. It is the inconsistencies that  may lead us to learning something new, not necessarily the umpteenth recitation of the general tendency. This is why we should pursue cases where race and poverty make little difference to school achievement. This is why we should take notice that the poor sometimes choose low fee private schools instead of free public schools. This is why we should investigate why some cultures appear to be more diligent. This is why we should consider the possibility that schools that include God in the curriculum may have more manageable classrooms. This is why we should notice that parents, once offered a choice of schools, do not vote to return to the ex ante status. And this is why we should focus on whether it is normal for all families, not just white families, to flee schools that are dangerous.


I would suggest that graduate schools of education end the segregation of research techniques; that they teach research techniques as one continuum and place as much importance on personal interview and observation as on causal inference. The more expensive and more precise techniques have not led to greater insight or to an amelioration of our main education problems. And for that reason, we should pause and rethink our approach. Our emphasis should be on two things only: insight and the political feasibility of improvement. Other things, such as the elegance of precision: those are of secondary importance.


Notes


1. Adapted from Heyneman (1997).


References


Coleman, J. S. (1988). Social Capital in the Creation of Human Capital. American Journal of Sociology, 94, 95-120.


Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. New York: Jossey-Bass.


Heyneman, S. P. (1976). A Brief Note on the Relationship between Socioeconomic Status and Test Performance Among Ugandan Primary School Children. Comparative Education Review, 20(1), 481-502.


Heyneman, S. P. (1977). Influences on Academic Achievement: A Comparison of Results from Uganda and More Industrial Societies. Sociology of Education, 11(2), 245-259.


Heyneman, S. P. (1979). Why Impoverished Children do well in Ugandan Schools. Comparative Education, 15(2), 175-185.


Heyneman, S. P. (1997). Jim Coleman: A Personal Story. Educational Researcher, 26(1), 28-30.


Heyneman, S. P. (2005). Student Background and School Achievement: What is the Right Question. American Journal of Education, 112(1), 1-9.


Heyneman, S. P.  (2013). The International Efficiency of American Education: The Bad and the Not-so-Bad News. pp. 279-302 in Heinz-Dieter Meyer and Aaron Benavot (Eds.) Pisa, Power and Policy: The Emergence of Global Educational Governance. Oxford, UK:  Oxford Studies in Comparative Education Symposium Books.


Heyneman, S. P. & Loxley, W. (1983). The Effect of Primary School Quality on Academic Achievement Across 29 High- and Low-Income Countries.  American Journal of Sociology, 88(6), 1162-1194.


Neumann, A. (2006). Professing Passion: Emotion in the Scholarship of Professors at Research Universities. American Education Research Journal, 43(3), 381-424.


Ogbu, J. (1991). Minority Coping Responses and School Experience. Journal of Psychohistory, 18(4), 433-456.


Ravitch, D. (2010). The death and life of the great American school system: How testing and choice are undermining education. New York: Basic Books.


Tucker, M. (Ed.) (2011). Surpassing Shanghai: An agenda for American education built on the world's leading systems. Cambridge, MA: Harvard University Press.


APPENDIX


List of sources for school choice


In favor of school choice


Peterson (21)


Peterson, P. E. (1995). A critique of the Witte evaluation of Milwaukee's school choice program. Center for Political Studies, Department of Government, Harvard University.


Greene, J., Peterson, P. E., & Du, J. (1996). School choice in Milwaukee.Public Interest (September).


Greene, J. P., Peterson, P. E., & Du, J. (1997). Effectiveness of school choice: The Milwaukee experiment. Program in Education Policy and Governance and Center for American Political Studies, Harvard University.


Peterson, P. E., Green, J. P., Du, J., & Hassel, B. C. (1998). The Effectiveness of School Choice in Milwaukee: A Secondary Analysis of Data from the Programs Evaluation. Learning from School Choice.


Peterson, P. E. (1998). School choice: A report card. Va. J. Soc. Pol'y & L.6, 47.


Peterson, P. E., Myers, D., & Howell, W. G. (1998). An Evaluation of the New York City School Choice Scholarships Program: The First Year.


Peterson, P. E., & Hassel, B. C. (Eds.). (1998). Learning from school choice. Brookings Institution Press.


Greene, J. P., Howell, W. G., & McCready, W. (1998). Initial findings from an evaluation of school choice programs in Washington, DC and Dayton, Ohio. Program on Education Policy and Governance, Harvard University.


Peterson, P. E., Myers, D. E., Howell, W. G., & Mayer, D. P. (1999). The effects of school choice in New York City. Earning and learning: How schools matter, 317-339.


Peterson, P. E. (1999). Vouchers and Test Scores. Policy Review93, 10-15.


Peterson, P. E., Howell, W. G., & Greene, J. P. (1999). An Evaluation of the Cleveland Voucher Program after Two Years.


Howell, W. G., & Peterson, P. E. (2000). School choice in Dayton, Ohio: An evaluation after one year. Program on Educational Policy and Governance, Harvard University.


Howell, W. G., Wolf, P. J., Peterson, P. E., & Campbell, D. E. (2000). Test-score effects of school vouchers in Dayton, Ohio, New York City, and Washington, DC: Evidence from randomized field trials. ERIC Clearinghouse.


Wolf, P. J., Howell, W. G., & Peterson, P. E. (2000). School choice in Washington, DC: An evaluation after one year. Program on Educational Policy and Governance, Harvard University.


West, M. R., Peterson, P. E., & Campbell, D. E. (2001). School choice in Dayton, Ohio after two years: An evaluation of the parents advancing choice in education scholarship program. John F. Kennedy School of Government, Harvard University.


Mayer, D. P., Peterson, P. E., Myers, D. E., Tuttle, C. C., & Howell, W. G. (2002). School choice in New York City after three years: An evaluation of the school choice scholarships program. Washington, DC: Mathematica Policy Research, Inc.


Howell, W. G., Wolf, P. J., Campbell, D. E., & Peterson, P. E. (2002). School vouchers and academic performance: Results from three randomized field trials. Journal of Policy Analysis and Management21(2), 191-217.


Peterson, P., Howell, W., Wolf, P. J., & Campbell, D. (2003). School vouchers. Results from randomized experiments. In The economics of school choice (pp. 107-144). University of Chicago Press.


Peterson, P. E. (2003). The future of school choice.


Peterson, P. E., & Howell, W. G. (2004). Efficiency, Bias, and Classification Schemes A Response to Alan B. Krueger and Pei Zhu. American Behavioral Scientist47(5), 699-717.


West, M. R., & Peterson, P. E. (2006). The efficacy of choice threats within school accountability systems: Results from legislatively induced experiments.The Economic Journal116(510), C46-C62.



Hoxby (19)


Hoxby, C. M. (1996). The effects of private school vouchers on schools and students. Holding schools accountable: Performance-based reform in education, 177-208.


Hoxby, C. M. (1998). What do America’s ‘traditional’forms of school choice teach us about school choice reforms. Federal Reserve Bank of New York Economic Policy Review4(1), 47-59.


Hoxby, C. M. (1998). Analyzing school choice reforms that use America’s traditional forms of parental choice. Learning from school choice, 133-156.


Hoxby, C. M. (1998). When Parents Can Choose, What Do They Choose? The Effects of School Choice on Curriculum. When Schools Make A Difference.


Hoxby, C. M. (1999). The effects of school choice on curriculum and atmosphere. Earning And Learning: how schools matter, 281-316.


Hoxby, C. M. (2000). Does Competition among Public Schools Benefit Students and Taxpayers?", American Economic Review, 90 (5): 1209-1238. 


Hoxby, C. M. (2001). How school choice affects the achievement of public school students. Choice with equity, 150.


Hoxby, C. M. (2003). School choice and school competition: Evidence from the United States. Swedish Economic Policy Review10(2), 9-66.


Hoxby, C. M. (2003). School Choice and School Productivity. Could School Choice be a tide that lifts all boats?. In The economics of school choice (pp. 287-342). University of Chicago Press.


Caroline M. Hoxby (editor). 2004. College Choices: The Economics of Where to Go, When to Go, and How to Pay for It. University of Chicago Press. 


Hoxby, C. M. (2004). Achievement in charter schools and regular public schools in the United States: Understanding the differences. Harvard University.


Hoxby, C. M., & Rockoff, J. E. (2004). The impact of charter schools on student achievement. Department of Economics, Harvard University.


Hoxby, C. M. (2004). A straightforward comparison of charter schools and regular public schools in the United States. National Bureau of Economic Research.


Hoxby, C. M. (2005). Competition among public schools: a reply to Rothstein (2004) (No. w11216). National Bureau of Economic Research.


Hoxby, C. M. (2006). School choice: The three essential elements and several policy options. Education Forum and New Zealand Association of Economists.


Hoxby, C. M., & Murarka, S. (2006). Comprehensive yet simple: Florida's tapestry of school choice programs. Reforming education in Florida: A study prepared by the Koret Task Force on K-12 education, 167-211.


Hoxby, C. M., & Murarka, S. (2006, September). Methods of assessing the achievement of students in charter schools. In Prepared for the National Conference on Charter Schools Research, Vanderbilt University, September(Vol. 28).


Hoxby, C. M. (Ed.). (2007). The economics of school choice. University of Chicago Press.


Hoxby, C. M., & Murarka, S. (2009). Charter schools in New York City: Who enrolls and how they affect their students' achievement (No. w14852). National Bureau of Economic Research.


Against school choice


Carnoy (14)


Carnoy, M. (1995). Is School Privatization the Answer? Data from Other Countries Help Burst the Voucher Bubble. American Educator19(3), 29-30.


Carnoy, M. "Is privatization through education vouchers really the answer? A comment on West." The World Bank Research Observer 12, no. 1 (1997): 105-116.


Carnoy, M. (1998). National Voucher Plans in Chile and Sweden: Did privatization reforms make for better education?. Comparative education review42(3), 309-337.


Carnoy, M., & McEwan, P. (1998). Is Private Education More Effective and Cost-Effective than Public?. The Case of Chile. Draft. Stanford University.


Rothstein, R., Carnoy, M., & Benveniste, L. (1999). Can Public Schools Learn from Private Schools? Case Studies in the Public & Private Nonprofit Sectors. Economic Policy Institute, 1660 L St, NW, Suite 1200, Washington, DC 20036.


Carnoy, M. (2000). School choice? Or is it privatization?. Educational Researcher29(7), 15-20.


McEwan, P. J., & Carnoy, M. (2000). The effectiveness and efficiency of private schools in Chile's voucher system. Educational evaluation and policy analysis22(3), 213-239.


Carnoy, M. (2001). School Vouchers: Examining the Evidence. Economic Policy Institute, 1660 L Street, NW, Suite 1200, Washington, DC 20036.

Benveniste, L., Carnoy, M., & Rothstein, R. (2003). All Else Equal: Are Public and Private Schools Differents?. Routledge.


Carnoy, M., & McEwan, P. (2003). Does privatization improve education? The case of Chile’s national voucher plan. Choosing choice: School choice in international perspective, 24-44.


Carnoy, M., Jacobsen, R., Mishel, L., & Rothstein, R. (2005). The charter school dust-up. Economic Policy Institute.


Carnoy, M., Jacobsen, R., Mishel, L., & Rothstein, R. (2006). Worth the price? Weighing the evidence on charter school achievement. Education1(1), 151-161.


Carnoy, M. (2007). Cubas Academic Advantage: Why Students in Cuba Do Better in School. Stanford University Press.


Carnoy, M., Adamson, F., Chudgar, A., Luschei, T. F., & Witte, J. F. (2007).Vouchers and public school performance: A case study of the Milwaukee parental choice program. Economic Policy Inst.


Rothstein (6)


Rasell, E., & Rothstein, R. (1993). School Choice: Examining the Evidence. Public Interest Publications, PO Box 229, Arlington, VA 22210.


Rothstein, R., Carnoy, M., & Benveniste, L. (1999). Can Public Schools Learn from Private Schools? Case Studies in the Public & Private Nonprofit Sectors. Economic Policy Institute, 1660 L St, NW, Suite 1200, Washington, DC 20036.


Rothstein, R. (2000). CAN SCHOOL CHOICE C DELIVER WHAT IT PROMISES? O. Restructuring education: innovations and evaluations of alternative systems, 65.


Rothstein, R. (2001). New federal roles in education. Publication of the Center on Education Policy< http://www. ctredpol. org/policy_papers/future_fed_role_rothstein. pdf.


Benveniste, L., Carnoy, M., & Rothstein, R. (2003). All Else Equal: Are Public and Private Schools Differents?. Routledge.


Carnoy, M., Jacobsen, R., Mishel, L., & Rothstein, R. (2005). The charter school dust-up. Economic Policy Institute.


Kreuger (2)


Krueger, A. B., & Zhu, P. (2004). Another look at the New York City school voucher experiment. American Behavioral Scientist47(5), 658-698.


Krueger, A. B., & Zhu, P. (2004). Inefficiency, Subsample Selection Bias, and Nonrobustness A Response to Paul E. Peterson and William G. Howell.American Behavioral Scientist47(5), 718-728. 


Hanushek (2)


Hanushek, E. A., Kain, J. F., Rivkin, S. G., & Branch, G. F. (2007). Charter school quality and parental decision making with school choice. Journal of Public Economics91(5), 823-848.


Hanushek, E. A., Kain, J. F., O'Brien, D. M., & Rivkin, S. G. (2005). The market for teacher quality (No. w11154). National Bureau of Economic Research. 


Neutral


C. Rouse (3)


Rouse, C. E. (1998). Private school vouchers and student achievement: an evaluation of the Milwaukee Parental Choice Program, Quarterly Journal of Economics, 113, 553-602.


Rouse, C. E., & Barrow, L. (2008). School vouchers and student achievement: Recent evidence, remaining questions (No. 2008-08). Working Paper, Federal Reserve Bank of Chicago.


Figlio, D. N., & Rouse, C. E. (2006). Do accountability and voucher threats improve low-performing schools?. Journal of Public Economics90(1), 239-255.






Cite This Article as: Teachers College Record, Date Published: January 31, 2014
https://www.tcrecord.org ID Number: 17400, Date Accessed: 12/3/2021 8:59:11 AM

Purchase Reprint Rights for this article or review
 
Article Tools
Related Articles

Related Discussion
 
Post a Comment | Read All

About the Author
  • Stephen Heyneman
    Vanderbilt University
    STEPHEN P. HEYNEMAN received his PhD in comparative education from the University of Chicago in 1976. He served the World Bank for 22 years. Between 1976 and 1984 he helped research education quality and design policies to support educational effectiveness. Between 1984 and 1989 he was in charge of external training for senior officials world wide in education policy. And between 1989 and 1998, he was responsible for education policy and lending strategy, first for the Middle East and North Africa and later for the 27 countries of Europe and Central Asia. In July, 2000 he was appointed professor of International Education Policy at Vanderbilt University. Current interests include the effect of higher education on social cohesion, the international trade in education services and the economic and social cost to higher education corruption.
 
Member Center
In Print
This Month's Issue

Submit
EMAIL

Twitter

RSS