Mixed Methods for Studies That Address Broad and Enduring Issues in Education Research

by Lois Weis, Margaret Eisenhart, Greg J. Duncan, Elizabeth Albro, Andrea Conklin Bueschel, Paul Cobb, Jacquelynne Eccles, Ruby Mendenhall, Pamela Moss, William Penuel, Robert K. Ream, Rubén G. Rumbaut, Finbarr Sloane, Thomas S. Weisner & James Wilson - 2019

This special report of the Mixed Methods Working Group is edited by Lois Weis, Margaret Eisenhart, and Greg J. Duncan.

REPORT: MIXED METHODS FOR STUDIES THAT ADDRESS BROAD AND ENDURING ISSUES IN EDUCATION RESEARCH

THE MIXED METHODS WORKING GROUP

The Mixed Methods Working Group (MMWG) included senior-level scholars and funders who use or support the use of multiple methods in education research. The group was convened to discuss guidelines for mixed-methods research that addresses broad and enduring educational problems in an increasingly diverse and unequal society by capitalizing on the complementary strengths of different methods.

BACKGROUND TO THE MMWG CONVENING

Over the past decade, research universities and funding agencies have vigorously encouraged research that addresses pressing and large-scale problems of education in an increasingly diverse and unequal society. It is widely accepted that such research requires research teams drawn from several disciplines and employing a range of research designs and methods. At least a handful of universities are pushing cluster hiring schemes that encourage cross-disciplinary and mixed-methods teams of researchers to build critical mass in vital fields of scholarship, including education.¹ Constructing and actualizing such broad-based teams demands that scholars leverage institutional support to pursue external federal, state, and/or private foundation funding and work across disciplinary and methodological boundaries to accomplish their goals.

Although there has been scholarly discussion of what mixed-methods research is or should be, limited attention has been paid to the ways in which such methods can be thoughtfully and rigorously employed in the service of broad-based projects that attack significant education issues, seek external funding, and strive for wide dissemination. Unfortunately, the value of multiple investigative methods in education research has, at times, been overshadowed by a Manichean debate pitting quantitative and qualitative research approaches against each other (Johnson & Onwuegbuzie, 2004). Although mixed-methods approaches have become increasingly popular, method syntheseswhere multiple research methods are used in truly integrated, coherent, and logical waysremain less common in education research. This type of research is difficult to conceptualize and implement. Many questions and constraints remain regarding research goals, methodological choices and coordination, constructive team development, and forms and venues for effective dissemination of research results.

The MMWG was convened to address these issues. We proposed two face-to-face working group meetings on the scope and possibilities of mixed-methods research for studying broad and enduring educational issues. Lois Weis, Margaret Eisenhart, and Greg Duncan approached the Spencer Foundation for support for this project. Specifically, we aimed to (1) consider what is and could be meant by mixed-methods research that goes well beyond qualitative work that employs a set of descriptive statistics or quantitative work that includes a few interviews; (2) address the value added of mixed-methods designs in the service of research projects aimed at contextualizing and addressing broad-based and significant educational problems; (3) detail the ways in which one method or one set of methods or research design (e.g., qualitative) can usefully leverage others (e.g., quantitative or experimental); (4) focus on issues specifically linked to preparing competitive proposals for funding that employ a mixed-methods design, with an eye toward providing guidance to potential grantees; and (5) focus on critical issues involved in publishing and disseminating mixed-method research in highly impactful venues. The meetings took place at the Spencer Foundation during a one-year period, 20142015.

The three conveners took responsibility for identifying and inviting all participants. Twelve participants were invited based on the following criteria: (1) research experience using multiple methods for studies of significant educational issues; (2) diversity of perspectives; (3) breadth of theoretical and empirical knowledge; (4) experience working on broad-based funded research teams; and (5) a record of publication in highly impactful venues. Three participants represented funders with interests in mixed-methods research in education. We were especially interested in attracting participants who were enthusiastic about working across methodological traditions and paradigms to think through the ways in which mixed-methods research can be leveraged in the service of team-based funded research tied to broad-based and pervasive educational problems. All those approached to be part of the group enthusiastically agreed to participate.

The goal of the MMWG was not to debate or come up with a singular definition of mixed-methods research, but rather to produce a document that would consider the utility of such research in the service of broad and significant research questions in education. There are rich literatures, with long histories, on mixed methods and interdisciplinary/transdisciplinary research (Bergman, 2008; Frodeman, Klein, & Mitcham, 2010; Greene, 2007; Repko, 2012; Tashakkori & Teddlie, 2010). We were seeking recommendations as to how empirical researchers could usefully address important and enduring educational problems using mixed-methods approaches. As such, our group addressed key features of successful mixed-methods research; challenges of proposing and conducting such research; ways to address such challenges; training in mixed-methods research; and issues of funding and publishing such work. To focus our discussion, we drew on examples of exemplary mixed-methods research suggested by all members of the MMWG (see the next section for lists and summaries of these examples).

A WORKING DEFINITION OF MIXED METHODS

The MMWG conceived of mixed methods more broadly than the specific tools used to collect information that represents the world we are trying to understandas important as those tools and expert uses of them are. We thought of mixed methods as encompassing broad uses of research concepts, theories, designs, and methods drawn from a wide range of disciplines, intellectual traditions, and research paradigms; as dynamic and integrated uses of methods that complement one another as they unfold over time and across levels and scales of a system; as guides to more comprehensive studies; as ways to collect, analyze, and integrate data creatively; and as ways to interpret and disseminate results to a wide range of audiences.²

MOVING BEYOND THE QUALITATIVEQUANTITATIVE BINARY

Integrating qualitative (text-based) and quantitative (number-based) methods is usually the way mixed methods are described. These two categories are themselves very broad, however, and the many kinds of qualitative methods and quantitative methods need to be specified. The term mixed methods is imprecise and can be confusing unless the particular qualitative and quantitative methods that are being used are carefully detailed. Further, the explicit or implicit division of methodological approaches into qualitative, quantitative, and mixed to describe, classify, and justify methods (or identify researchers) is overly simplistic for analytic and pedagogic purposes. Importantly, methods are not independent of the disciplinary and theoretical traditions in which they have been produced, a fact that the division into qualitative and quantitative obscures.

Further, research methods should not be chosen without regard for the research questions at issue (Johnson & Onwuegbuzie, 2004). Some methods are particularistic, useful for capturing a part of some phenomenon; others are holistic, attempting to capture the whole context or situation (Yoshikawa, Weisner, Kalil, & Way, 2008). Methods can be naturalistic, to unobtrusively capture ordinary human activity, or experimental, which control human activity to understand impacts on specific outcomes. Some methods are useful in understanding processes, and others in analyzing variables (Maxwell, 2011). Anthropologists have categorized methods as experience-near (representing the voices, intentions, meanings, and local rationality of people in local settings) and experience-distant (representing the meanings or concepts used by specialists to describe, categorize, or theorize other peoples experiences; Geertz, 1980). The affordances of methods given research questions, rather than their designation as qualitative or quantitative, should guide their use in a research design.

KEY FEATURES OF EFFECTIVE MIXED-METHODS RESEARCH

Fruitful Combinations

Over the years, education researchers have employed various techniques to investigate what goes on in schools and classroomstechniques including strictly controlled experiments, longitudinal surveys, and meta-analyses summarizing effects found across numerous studies. Additionally, micro-analyses of classroom discourse, in-depth interviews, and participant observations have become common research approaches. Education research is not wanting for methodological diversity. The fruitful combination of these and other methods is a hallmark of effective mixed-methods research. Methods can complement one another to deepen understanding or challenge one another by illuminating taken-for-granted assumptions and limitations (Moss & Haertel, 2016). They can corroborate some findings, interrogate and/or elaborate on others, and help to initiate new ideas not foreseen in the initial study design. The value of mixed-methods research for addressing broad-based projects that attack enduring education problems lies in their ability to draw on the unique strengths of varying methods so as to investigate these problems from a variety of perspectives, in a variety of contexts at potentially different points in time, and at a variety of levels (or scales). Yet collaboration and bidirectional problem solving among researchers employing different methods do not routinely occur, despite growing evidence that they should.

Integration and Iteration

A key feature of effective mixed-methods research is the ability to organizeintegrate, coordinate, or juxtaposethe findings from different methods. It may be fairly straightforward to add survey results to a participant observation study in order to assess generalizability or to add findings from interviews to an experimental study with the purpose of interpreting statistical outcomes. However, because different methods derive from different perspectives, paradigms, and research questions, they often produce disparate or inconsistent findings. When findings vary by method, mixed-methods researchers must figure out how to make sense of all the findings together and represent them to relevant audiences. This effort demands a high level of team interaction and a shared attitude of methodological and theoretical flexibility as a program of research moves forward (see the section on multidisciplinary teams that follows). A special power of mixed-methods research in the service of addressing broad and enduring educational problems is to successfully integrate, coordinate, or juxtapose findings derived from multiple methods in ways that expand and deepen understanding of problems.

The iterative use of methods is another hallmark of effective mixed-methods research. By iterative use, we mean that findings from one study (e.g., a controlled experiment or an ethnography) become the impetus for subsequent studies that both incorporate and move beyond the initial study.³ Subsequent studies may extend the reach, clarify the theory, or explore the unanticipated results of single studies. For example, unexpected findings from a large-scale quantitative study of teacher effectiveness might lead to in-depth studies of a few classrooms either drawn from the larger study or strategically chosen to illuminate the mechanisms through which previously identified outcomes are produced in classrooms and schools. Additionally, participant observation and narrative interviews can enable investigators to understand how to interpret findings from quantitative studies and develop more useful measures for future research. Also, positive outliers from a quantitative study can become the focus of case (or cross-case) analyses that aim to enable researchers to understand underlying processes (see Maxwell, 2011).

Iterative uses of mixed methods are especially valuable in longitudinal research, wherein various methods must often be used to follow individuals or groups over time. For example, as children grow up and move from context to context, different methods may be required to sensitively investigate their developing lives. Such mixed-methods studies, which may be undertaken by individual scholars or research teams, can be indispensable for studies across the life course. Design-based studies also may depend on iterations of multiple methods. In design-based research, interventions (e.g., in a classroom) are proposed in accord with a particular theory (e.g., a theory of learning), and their implementation and effects are studied in multiple ways in situ. Findings produced in one setting at one point in time are then assessed in terms of the theory. Findings that are inconsistent with the theory lead to revisions in practice or revisions of the theory, which are then restudied in situ. As theory and practice are refined over time, various methods may be employed to assess them. (Integration and iteration are further exemplified throughout the compilation of exemplary mixed-methods studies later in this article.)

Programs of Study

In many cases, mixed methods are used most effectively in programs of research, as well as in individual studies. There can, of course, be great analytical value in using single methods, or exclusively qualitative or quantitative methods, for particular studies, or a series of studies in which only certain methods are available. However, sustained research programs that focus on large-scale education issues must inevitably consider numerous geographical sites, multiple timescales, and nested complex systems. The complexity of large-scale programs of research makes them ideal candidates for mixed-methods research. Such programs of research are, as noted earlier, often iteratively built, wherein additional lines of investigation are added as research proceeds over time. Although mixed-methods studies are sometimes conceived from the outset, broad programs of research frequently add more components as the research proceeds.

CHALLENGES OF MIXED-METHODS RESEARCH AND PROPOSED SOLUTIONS TO SUCH CHALLENGES

Constituting and Coordinating Multidisciplinary Teams

Although mixed-methods and multidisciplinary teams are increasingly valued as a way to address large-scale questions of educational importance, constituting such teams poses a challenge. Investigators tend to develop in-depth expertise in particular theoretical and methodological perspectives. Those who lead and become part of mixed-methods and multidisciplinary teams must exhibit an intellectual and personal flexibility. They must also see the value of other methods and perspectives and develop a willingness to work across these methods and perspectives in the service of broad-based research questions. In all likelihood, one or two principal investigators (PIs) will initially conceptualize a large mixed-methods and interdisciplinary project, in conversation with a set of investigators who exhibit methodological and intellectual strength in relevant areas. The goal is not to undermine the value of one methodological or disciplinary perspective, but rather to recognize that approaching questions from a mixed perspective has the potential to expand the knowledge base in key areas. It is important that PIs continue to have a sense of the overall project, as well as all component parts, and be able to coordinate these key components as the group moves forward.

Funding

Although many funders limit the scope of the methodologies they will consider, a considerable number are interested in large-scale mixed-methods research designs in the service of interdisciplinary research questions of importance. Those scholars who propose such research must take careful account of the ways in which one method leverages the use of a second or third method and make explicit statements to this effect in proposals for funding. Employing additional methods cannot be seen as an add-on with no compelling purpose. For instance, if a large-scale survey and in-depth interviews are proposed, a thoughtful explanation should be provided regarding how each method complements the other in addressing the research problem, as well as what empirical insights will be gained by employing each method. Justifications can readily be found in existing scholarly research on the topic of interest and expected findings from the proposed research.

When seeking funding for mixed-methods projects, those who are leading the project, particularly PIs, must be cognizant of the fact that adding a method often involves adding an investigator, and this creates additional costs for the study. Generally speaking, the amount of research funding available within particular funding agencies has remained constant, making it difficult for investigators to constitute larger teams. This has potential implications for funding agencies because they may wish to rethink funding in such a way as to accommodate mixed-methods projects in particular instances (for example, setting aside some money for larger mixed-methods teams along the lines outlined here).

Training Programs in Mixed Methods

Training in research methods in doctoral programs usually involves courses in quantitative research and in qualitative research. Students are often expected to develop expertise in one of these methodological categories so that they can formulate a research question, collect data that answer their questions, and produce a dissertation. In some programs and universities, students are expected to develop expertise in one method, with some training in another type. For example, quantitatively focused individuals might be expected to take a one- or two-semester course on ethnographic methods, and qualitatively focused students would be expected to do a year of statistical methods. This is not optimal mixed-methods training, in our view.

As an alternative to traditional training, introductory research methods courses might focus on the development of students research agenda and the career implications of experience with multiple research methods. Training in specific research methods (by research agenda) would logically follow. In this way, students would learn to see methods as useful to particular kinds of questions, and they would be exposed to the nature of research wherein multiple methods can be employed in the service of developing a program of research. This training approach seems particularly critical for producing effective mixed-methods researchers.

While existing training programs largely encompass coursework aimed at intense technical training in one method, problem-centered programs in mixed-methods training can fruitfully be designed to build on and expand expertise in methods across the familiar quantitative/qualitative divide. Such training programs would emphasize the importance of research questions in determining the selection of research methods that can be used to address them. Such an approach, while acknowledging that questions are shaped in part by the methods known, underscores the fact that methods should address the research question(s) asked. At the conclusion of mixed-methods training, the trainees should be able to identify possible points of entry into collaborative projects involving multiple methods. As noted earlier, although courses for graduate students in research methods are available in many universities, such training, especially in schools of education, is not necessarily linked to specific research questions that might be collaboratively pursued over time and organized into a coherent program of research. Above and beyond traditional methods courses, methods training linked to specific research questions is more likely to produce future scholars who can envision and execute broad-based mixed-methods research and multidisciplinary research teams. Such problem-centered approaches to research can also be encouraged by apprenticeships for graduate students on multimethodological problem-focused research teams.

Mixed methods can also be promoted at the dissertation stage if students are encouraged to formulate dissertation research questions that require designs incorporating mixed methods and if dissertation committees are composed of faculty able to provide the needed expertise. The value of such dissertations can be enhanced if students are asked to reflect on the synergies generated by the mixing of methods.

To further encourage and strengthen mixed-methods research, funders could provide targeted postaward support, with the intention of bringing together grantees focused on a particular problem area who could benefit from considering mixed methods or from collaborating with scholars with different methodological strengths. With such support, sets of funded research teams could come together around particular problem areas to explore new methods, improve their skills, and learn from one another across projects. Researchers, postdoctoral scholars, junior faculty, graduate students, and undergraduate students associated with such projects could broaden their perspectives, gain additional training, and establish cross-disciplinary contacts.⁴

Professional Identity

Given current training, scholars are most likely to identify themselves as aligned with a particular method. A professional identity for mixed-methods researchersmore specifically, an identity for someone who has the training and ability to work on mixed-methods teamsshould be encouraged. By this we mean that graduate students should be prepared to collaborate on problem-focused mixed-methods teams. In other words, they should have some sense of the affordances of alternative methodologies and the ways in which varying methods can work together to address broad and enduring issues in education research.

Our goal here is productive collaboration across differences, not necessarily identical methodological preparation in the sense that all students would be expected to take all available courses linked to each major method. We continue to need scholars with deep expertise in particular methodologies. Scholars with a mixed-methods identity are those who can take such deep expertise and move comfortably across methodological boundaries as they work with others.

A mixed-methods identity would signal the kind of linguistic competence, comfort with and understanding of research logics, and familiarity with the precise vocabulary of various research designs that maximize the contributions of different methods. Researchers who are deeply and rigorously trained to pursue a problem-driven research agenda rather than a single study or a single method are those likely to work most productively on mixed-methods teams. Laying the groundwork through training that fosters mixed-method professional identity can facilitate the kind of mixed-methods teamwork and habits of mind necessary for large-scale interdisciplinary research on questions of importance in education.

Vetting and Evaluating Mixed-Methods Articles and Proposals

Editors of education research journals and funding agency program officers find it difficult to both advise researchers who wish to submit work that draws on multiple methods and identify reviewers with appropriate expertise to evaluate these submissions. By way of example, a mixed-methods proposal or submitted article might draw on multilevel statistical models and in-depth interviewing wherein few (or no) individual reviewers are likely to be able to adequately assess the merit of all aspects of the article or proposal. So too, a mixed-methods proposal or submitted article encompassing multimethodological, multilevel, multi-interacting contexts would pose challenges with regard to appropriate reviewers. As with building mixed-methods teams more generally, the review process must capitalize on individual reviewers strengths while simultaneously employing a range of reviewers with complementary strengths and editors capable of guiding authors through the at times conflicting reviewer advice. Having more researchers with mixed-methods sensibilities and credentials could lessen this problem. Clearly, careful evaluation of each method and of one method as linked to a specific research question must be sustained. However, those with mixed-methods expertise would be more likely to understand the value added of a mixed-methods approach while paying careful attention to the aspects of the article or proposal that are aligned with their own high-level expertise.

In addition, funders express concern that many mixed-methods proposals do not make clear how the various methods are connected to the research questions, the conceptual framework, theory, or the analysis and dissemination proposed. Journal editors have similar concerns. Mixed-methods proposals and articles must give careful attention to how all elements of the study fit together and will be or are coordinated. Investigators must carefully describe each method and the forms of analysis employed. Importantly, one method should not appear to be a mere add-on in an otherwise single-method study. The value of a mixed-methods approach should be apparent in the research questions, the framework or theory used, the choice of methods and analysis, and, in the case of an article, the findings and conclusions.

THE FUTURE OF MIXED-METHODS RESEARCH

The challenging education issues of today require sophisticated theorizing and the collection of complex data using mixed methods. Using these tools to capture educational phenomena, and the forces driving them, strengthens education policy and its outcomes. As a global leader, the United States must find ways to train the next generation of scholars to create mixed methods, broad-based research agendas that can serve as models to the world.

The Mixed Methods Research Working Group participants included:

Lois Weis, SUNY Distinguished Professor, University at Buffalo

Margaret Eisenhart, Distinguished Professor, Bob and Judy Charles Endowed Chair of Education, University of Colorado Boulder

Greg J. Duncan, Distinguished Professor, University of California, Irvine

Elizabeth Albro, Commissioner for the National Center for Education Research at the Institute of

Education Sciences

Andrea Conklin Bueschel, Associate Vice PresidentProgram, Spencer Foundation through 2016; Associate Provost for Strategy and Policy, Northwestern University

Paul Cobb, Professor, Vanderbilt University

Jacquelynne Eccles, Distinguished Professor, University of California, Irvine

Ruby Mendenhall, Associate Professor, University of Illinois at Urbana-Champaign

Pamela Moss, Professor, University of Michigan

William Penuel, Professor, University of Colorado Boulder

Robert K. Ream, Associate Professor, University of California, Riverside; Associate Program Officer, Spencer Foundation 20132015

Rubén G. Rumbaut, Distinguished Professor, University of California, Irvine

Finbarr Sloane, Program Officer, National Science Foundation

Thomas S. Weisner, Distinguished Professor Emeritus of Anthropology, Departments of Psychiatry and Anthropology, University of California, Los Angeles

James Wilson, Program Director, Russell Sage Foundation

Acknowledgments

The Mixed Methods Research Working Group was made possible by a research grant from the Spencer Foundation to Lois Weis, University at Buffalo; Margaret Eisenhart, University of Colorado Boulder; and Greg Duncan, University of California, Irvine. The work of the Mixed Methods Working Group was greatly facilitated by Rachel Dominguez, PhD student at the University at Buffalo. Thanks to Rachel for her hard work over the duration of the project. Additional thanks are given to Doris Fischer and Annie Brinkman at the Spencer Foundation.

Notes

1. See, for example, https://provost.ucr.edu/cluster-hiring/.

2. Because mixed methods is a familiar term and has a history within our broad interdisciplinary field, we use the term mixed methods rather than multiple methods for the purposes of this document.

3. The term iterative signals a pragmatic approach to mixed-methods research insofar as the pragmatic approach systematically integrates both quantitative and qualitative methods on at least two levels. First, data collection is staggered, often beginning with an initial field study followed by a survey wave and then a confirmatory field study (reanalysis of the survey data is likely to include additional variables emerging from the subsequent field study). Second, field study variables are keyed to survey variables to determine the precise degree of overlap and corroboration between the two.

4. The William T. Grant Foundation supported such an effort, bringing together scholars from its program on research utilization to identify dilemmas and share strategies for integrating data from different sources in order to draw inferences about research use. A particular strength of this effort was that it brought scholars together from different disciplines, including education, public health, and social work.

References

Bergman, M. M. (2008). Advances in mixed methods research: Theories and applications. London, England: SAGE.

Frodeman, R., Klein, J. T., & Mitcham, C. (2010). The Oxford handbook of interdisciplinarity. Oxford, England: Oxford University Press.

Geertz, C. (1980). Blurred genres: The refiguration of social thought. The American Scholar, 49(2), 165179.

Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 1426.

Maxwell, J. A. (2011). A realist approach for qualitative research. Thousand Oaks, CA: SAGE.

Moss, P., & Haertel, E. (2016). Engaging methodological pluralism. In D. Gitomer & C. Bell (Eds.), Handbook of research on teaching (pp. 127248). Washington, DC: American Educational Research Association.

Repko, A. F. (2012). Interdisciplinary research: Process and theory. Thousand Oaks, CA: SAGE.

Tashakkori, A., & Teddlie, C. (Eds.). (2010). Sage handbook of mixed methods and behavioral research.

Thousand Oaks, CA: Sage.

Yoshikawa, H., Weisner, T. S., Kalil, A., & Way, N. (2008). Mixing qualitative and quantitative research in developmental science: Uses and methodological choices. Developmental Psychology, 44(2), 344354.

 

 

 

Cite This Article as: Teachers College Record Volume 121 Number 10, 2019, p. 1-16 https://www.tcrecord.org ID Number: 22741, Date Accessed: 1/28/2022 10:46:39 PM Purchase Reprint Rights for this article or review