Reflections on the Promise of Mixed-Methods Research in Education
by Carola Suárez-Orozco - 2019
This commentary is part of "Mixed Methods for Studies That Address Broad and Enduring Issues in Education Research," edited by Lois Weis, Margaret Eisenhart, and Greg J. Duncan.
As the authors of this important contribution clearly articulate, there is little doubt that the complex and enduring educational dilemmas of our time require the promise that mixed-methods approaches offer. The reasons to deploy mixed methods are quite obviousespecially if we are to understand why what we are trying to accomplish matters for whom (Hay, 2016; Mertens, 2015). In this commentary, I offer some thoughts based on a quarter century of mixing methods.
THE DEMOGRAPHIC IMPERATIVE
The authors articulate that mixed-methods work is important if we are to address an increasingly diverse and unequal society. Mertens advocates that transformative mixed-methods approaches allow for the development of trusting ethical relationships, and the joint, iterative exploration of issues that are meaningful to the community (Mertens, 2012). I would further suggest that, given the growing demographic diversity of our student population, it is deeply problematic from the social science point of view not to engage in mixed-methods research. Here is why.
Today, more than a quarter of our children, in most postindustrial nations, have an immigrant parent; thus, these students are growing up in culturally and linguistically diverse families. The classic instruments and assessment strategies, developed for mainstream and English-dominant students, often miss the experiences of these students and their parents because these approaches were developed through an entirely different lens (American Psychological Association [APA], 2012). Further, assessment items developed with mainstream or language-dominant norm groups can (and often do) lead to distorted interpretations that lead to miscategorizations and underrecognition of skills (APA, 2012). To ethically conduct linguistically and culturally relevant and competent assessment and research with students from these backgrounds, we must engage in an iterative practice of either adapting old tools or developing new ones that capture these students repertoires of lived reality and skills. This requires sequencing mixed methods, including a qualitative phase of engaging learning around cultural understandings and practices before moving to quantitative adaptations (or development) of assessments that include both student and parent points of view. Failing to do so can lead to a form of institutionalized racism (Jones, 2000).
MULTILINGUALISM AS METAPHOR FOR TEAMWORK
A current challenge in the field is that few of us (old or new scholars) have been systematically trained to be mixed-methods researchers. Most of us are thoroughly socialized into theoretical epistemologies of one discipline to use a set of methodological tools. While we may learn the language of a new field, like most second-language speakers, we are likely to speak with an accent (sometimes a heavy one) and will tend to favor our dominant disciplinary language. To work optimally in mixed-methods research, we should plan on working in collaborative teams that engage experts with a variety of skillsacross disciplines and across populations and/or topic areas.
For optimal work to be done, each team member must, at a minimum, have receptive knowledge of other research team methodological languages in order to follow the train of research conversation. We mixed-methods researchers need to continuously push ourselves to improve our methodological skills and to participate in active code-switching conversations in which no tedious interruptions are necessary to maintain the flow of conversation and creative riffing of ideas. Further, there must be a clear understanding of where the limits of our knowledge begin and end. When is it time to bring in an expert in what domain? What needs back-translation for accuracy? When is an entirely new language (or dialect/expertise/instrument) required in the planning or analysis stages? And, importantly for dissemination, subject/method experts with no accent must write select sections to pass reviewers critical lenses. Last, someone who is quite bilingual must polish final products to ensure that language is not jarringly disconnected and speaks to all audiences.
A major roadblock for mixed methods has been the issue of publication in high-impact venues. The reasons for this challenge have been multifold. In the course of mixed-methods research, massive volumes of data are typically collected. A challenge, then, becomes how to concisely tell the story the data have to tell in a succinct and yet valid, trustworthy, legitimate (Onwuegbuzie & Johnson, 2006), or transparent (Levitt et al., 2018) manner (depending on the disciplinary nomenclature). Books can be the more appropriate venue in which to lay out the full narrative of findings, though in some disciplines, books are not the valued coin of the realm.
A sensible strategy for many, then, is to parse out segments of the full mixed-methods data into various articles. Sometimes the data can be presented as simply a quantitative segment of the data, and at others as a qualitative portion; in such cases, there are specialized journals for each with reviewers who understand how to assess the type of research question, methods, analytic strategies, and appropriate conclusions that can be drawn. In other cases, the compelling elegance of the mixed-methods data is what author(s) wish to roll out. It may indeed be precisely what reveals a new finding in the most novel and compelling manner.
Yet publishing mixed-methods narratives in journals has, over the last decades, been a challenge, both because of increasingly tight journal word constraints and the limitations of reviewer training noted earlier. Most reviewers are well trained in one method but not others and thus impose their disciplinary and methodological lenses on the work they review, often without knowingly recognizing those biases. Further, the field has been constricted by its publishing standards. The American Psychological Association manual (APA, 2010), a standard in the field for many journals, for example, has privileged a quantitative paradigm. It has heretofore not provided much in the way of guidance for authors to prepare or reviewers to systematically review qualitative or mixed-method manuscripts. In an important step for the field, the APA convened a task force to develop and recommend explicit guidelines for both qualitative and mixed-methods research (Levitt et al., 2018). These will be the basis for the new APA manual soon to be released, providing explicit guidelines for editors to offer to both authors and reviewers moving forward.
In sum, the realities of doing meaningful mixed-methods work require significant (and continuous) training, flexibility, collaboration, and clear communication strategies. Mixed-methods team research is ideally suited to address pragmatic problem-centered issues with diverse learners like those that we encounter in the field of education. It necessitates social scientists who are well trained in more than one disciplinary and methodological approach, who are comfortable recognizing the limits of their training, and who effectively work collaboratively with others, laying out a clear transparent audit trail (Levitt et al., 2018). By working together iteratively, with genuine curiosity, toward a shared agenda to solve the critical educational issues of our time, I am optimistic that we can and will make great strides.
American Psychological Association. (2010). Publication manual of the American Psychological Association. Washington, DC: Author.
American Psychological Association. (2012). Crossroads: The psychology of immigration in the new century. APA Presidential Task Force on Immigration. Washington, DC: Author. Retrieved from http://www.apa.org/topics/immigration/report.aspx
Hay, M. C. (2015). Methods that matter: Integrating mixed methods for more effective social science research. Chicago, IL: University of Chicago Press.
Jones, C. P. (2000). Levels of racism: A theoretical framework and a gardeners tale. American Journal of Public Health, 90(8), 12121215.
Levitt, H. M., Bamburg, M., Creswell, J. W., Frost, D., Josselson, R., & Suárez-Orozco, C. (2018). Journal article reporting standards for qualitative primary, qualitative, meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board Task Force Report. American Psychologist, 73(1), 2646.
Mertens, D. M. (2012). Transformative mixed methods: Addressing inequities. American Behavioral Scientist, 56(6), 802813.
Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13(1), 4863.