This article explores how educational researchers can use meta-analysis to “power-up” the findings of their existing, small-scale qualitative research studies. By triangulating data from three independently conducted studies of academically at-risk college students, this research contests “time-to-degree” as a valid criterion for measuring academic success in college.
This chapter presents an introduction to design-based implementation research (DBIR). We describe the need for DBIR as a research approach that challenges educational researchers and practitioners to transcend traditional research/practice barriers to facilitate the design of educational interventions that are effective, sustainable, and scalable. We examine antecedents to DBIR, including evaluation research, community-based participatory research, design-based research, and implementation research. The four core principles of DBIR are explained: (1) a focus on persistent problems of practice from multiple stakeholders’ perspectives; (2) a commitment to iterative, collaborative design; (3) a concern with developing theory and knowledge related to both classroom learning and implementation through systematic inquiry; and (4) a concern with developing capacity for sustaining change in systems. We close with an overview of the chapters contained in this NSSE Yearbook on DBIR and explain how each chapter contributes to the overall development of the DBIR approach.
This chapter reviews four projects that reflect the principles of design-based implementation research (DBIR) in an effort to highlight a range of relevant theoretical and methodological perspectives and tools that can inform future work associated with DBIR.
Applied researchers, whether working with the framework of design-based research or intervention science, face a similar implementation challenge: practitioners who enact their programs typically do so in varied, context-specific ways. Although this variability is often seen as a problem for those who privilege fidelity and standardization, we argue for the advantages of researcher-practitioner collaborations that encourage local adaptation and ingenuity. We develop this argument for adaptive interventions by discussing two design-based research projects, Critical Civic Inquiry (CCI) and Science Literacy through Science Journalism (SciJourn), which create opportunities for youth to develop civics and science literacy respectively. CCI and SciJourn aim to build curricula that will travel to new schools and districts, but not through standardization. This is a delicate combination: the program must be flexible enough to enable productive adaptation, without being so protean that practitioners’ implementations lack substantive commonalities. We present two cases that show how project designers have sought to distinguish between invariant principles that define the intervention and heterogeneous practices that vary across sites. The cases also show how the model has improved when teachers can adapt it to their institutional context and when teachers and researchers establish social norms that encourage dialogic interactions.
This chapter focuses on how researchers and practitioners negotiate the focus of their joint work within design-based implementation research (DBIR). Studying and facilitating successful negotiation of the problems that become the focus of work and the search for solutions is important for developing DBIR, because of its commitment to focusing on persistent problems of practice from multiple stakeholders’ perspectives. Case studies of two different research–practice partnerships provide a context for exploring two different perspectives on negotiation. In one case study, the notion of partnerships as forms of cultural exchange across institutional boundaries that differ with respect to goals, norms, and practices is used to analyze a design partnership focused on repurposing curriculum units in elementary science. In the second case study, the concept of framing as developed in social movement theory is used to illuminate issues of status and authority within a partnership between a district and researchers. The chapter concludes by describing the contributions of each perspective to an understanding of how teams jointly negotiate the focus of their work and by providing some recommendations for how teams can do so successfully.
This chapter describes a partnership with four urban districts that aimed to develop an empirically grounded theory of action for improving the quality of mathematics instruction at scale. Each year, we conducted a data collection, analysis, and feedback cycle in each district that involved documenting the district’s improvement strategies, collecting and analyzing data to assess how these strategies were being implemented, reporting the findings to the district, and making recommendations about how the strategies might be revised. We distinguish between two distinct levels
Educational interventions typically are complex combinations of human actions, organizational supports, and instructional resources that play out differently in different contexts and with different kinds of students. The complexity and variability of outcomes undermines the notion that interventions either “work” or “don’t work.” Under the design-based implementation research (DBIR) model, the implementation of an intervention in particular settings is itself an object of research and a critical part of understanding how to scale an intervention without diluting its effectiveness. In this chapter, we compare the approach to evidence implicit in the defining features of DBIR to the prevailing evidence standards for educational research promoted by national policy. Our aim is to provide a frame for knowledge building within DBIR that draws from the strengths of both design-based research methods and research designs that permit causal inference about program impacts. Moreover, we endeavor to show how DBIR challenges current thinking about what counts as credible research. We conclude by considering the ways in which DBIR is a departure from much educational research in terms of what it means to conduct research that is useful and usable in education settings.
Design-based implementation research offers the opportunity to rethink the relationships between intervention, research, and situation to better attune research and evaluation to the program development process. Using a heuristic called the intervention development curve, I describe the rough trajectory that programs typically follow as they evolve, and argue that research design considerations and methodological choices are best made in consideration of where interventions are along this curve. Further, I contend that, as programs develop, situational influences play a major role in their evolution and consequently require increased attention to design and methodological considerations. By viewing research as an integral part of a program’s development, by making design and methodological choices in consideration of where programs are in their development, and by considering that the situation in which programs evolve as a potential source of change in the nature of the program itself, we alter fundamental perspectives on how research can best contribute to the steady work of building robust programs for educational improvement.
Education researchers are increasingly working in practice-based partnerships in order to direct their research efforts toward important problems of practice. We argue for the creation of an infrastructure to support routine and sustained interaction among researchers, practitioners, and designers in order to make partnership efforts more efficient and effective over time. We describe SERP’s efforts to initiate such an infrastructure through the creation and operation of “field sites”—long-term research, development, and implementation partnerships with school districts. We describe principles for field site operation and for product design that are emerging through our “learning by doing” approach. And we argue that the field site partnerships have produced work that is fundamentally different in character and content because of the SERP rules of engagement. SERP work is also more coherent because of its interdisciplinary character, and the sophistication of the partnership’s work grows over time as related lines of work become integrated and as those engaged in sustained collaboration learn from each other.
In this chapter, we describe the evolution of an intermediary organization, the Institute for Learning (IFL) at the University of Pittsburgh, devoted to improving teaching and learning in some of America’s largest and most challenged school districts. Over the nearly 20 years of IFL’s life, we have learned that design-based researchers have to move beyond classrooms and focus on entire school systems if they hope their innovations will scale and sustain. Specifically, we believe the best way to develop capacity for sustaining innovation is through the careful design of practices intended to provide an ongoing forum for mutual engagement among actors from various levels of the education system. These practices, we have found, provide a productive way for actors from different levels of the system to understand and support the local activity of other levels.
This chapter discusses frameworks and conceptual lenses that help orient design-based implementation research (DBIR) work to the types of infrastructure required for success, while contributing to theories about the processes of educational improvement. Such infrastructures can be conceived as a framework: a set of interconnected elements that facilitate the integrated development of an initiative, provide a continuing narrative for this development, create shared responsibility for its implementation, and facilitate its sustainability. The chapter posits that several types of infrastructure are necessary for successful, sustainable DBIR, including conceptual frameworks that attend to issues of scale, human capacity, technical support, policies, organizational learning, and long-term funding. In addition, the chapter discusses the roles that feedback loops, intermediary organizations, and multiple timescales play in systemic educational improvement, and describes how attention to these different infrastructural needs can lead to sustained improvements and support their evolution over time. The chapter also comments on theoretical and practical advances in social policy that often are not integrated into learning sciences research but are crucial for evolutionary educational improvement, including using organizational learning as a measure of progress, attending to social feedback loops and networks inside and outside of schooling, and utilizing the tools of policy informatics.
The research project involved four campuses that chose to integrate the arts into tested subject areas. Digital storytelling technology was combined with the narrative inquiry research method to produce ‘digital narrative inquiries’—16-20 minute multimedia representations of each school’s experience of its change initiative. Video clips excerpted from these larger productions were then used to discuss the affordances and constraints of using the ‘digital narrative inquiry’ representational form to share research findings arising from innovative educational practices.
This investigation derives four quantifiable measures of school quality and subsequently tests them in an empirical model that relates school-level inputs to school-level outputs. By conducting this analysis on a longitudinal, school-level data set of all 175 elementary schools within the School District of Philadelphia, this research provides new evidence that a range of school-level inputs has a significant relationship with school quality.
This article introduces two-dimensional video-based animations of fictional classroom interactions as a new kind of video image for representing classrooms. The authors suggest that this addition to the existing repertoire for representing practice can support conversations about tactical and strategic dimensions of the work of teaching in ways that overcome some of the limitations of usual video footage.
I was invited to give the annual lecture that honors Lawrence Cremin, the historian of American education at Teachers College, Columbia University. To pay tribute to the way in which Cremin used an academic discipline to bring rigor and depth to educational research, I described the way in which I used an academic discipline—linguistics and its varied tools of discourse analysis—to conduct research at the College.
As education librarians seek to collaborate with preservice teacher preparation programs, they need to apply informatics principles to optimize the library’s ultimate impact on student achievement. Specifically, education librarians need to examine several levels of information processing systems: student, faculty, program, institution, and government entities. Furthermore, education librarians need to identify the conditions or environments of these information systems because the infrastructure, available resources, and knowledge base all impact student learning.
This article discusses user information behavior though analysis of demographic factors, academic disciplinary characteristics, and the nature of educational research. Understanding elements of user information behavior will inform the development of a system of education informatics.
This essay introduces the issue, Learning Research as a Human Science.
The author suggests that questions in educational research are often about constitution, and that to answer such questions we want a methodology that goes beyond randomized clinical trials and customary qualitative research methods. The author focuses on the contributions of ethnographic fieldwork to research on constitution, though interviews and detailed interaction analysis are also important components.
In the spirit of deepening our understanding of the social conditions of everyday uses of mathematics, the authors studied 20 diverse families with a middle school child by interviewing family members together at home about their occasions of mathematics use.
This chapter highlights two important challenges in studying identity in learning contexts from a human science perspective. The first challenge is integrating different perspectives and potential contradictions in accounts of identity. The second is considering both presented and authentic selves in accounts of identity. Both of these challenges stem from a concern with understanding the complexity of identity in learning contexts and with capturing critical nuances in theoretical accounts of identity.
This chapter discusses a teacher–researcher partnership oriented toward phronesis, or wise action used to solve practical problems. The three practical problems the authors emphasize are: (1) How did they use their work together to improve teaching and learning? (2) How did they relate to each other in the work? and (3) How did they organize their time and resources to do the work, given their organizational settings and constraints?
This chapter examines productive tensions that arise for research and learning when university researchers collaborate with young people to identify, study, and act on relevant social problems. These tensions are explored through a discussion of an intergenerational and interracial youth participatory action research (YPAR) project to study the impact of a high school closure on students. The chapter identifies tensions and dilemmas experienced by participants, including the author, and discusses insights that these generate for doing research in partnership with youth.
This chapter presents a discussion of obstacles and impediments to and immobilities of a study of queer street-involved youth. The authors’ specific purpose is to identify and consider several specific “blockages” that the study brought to light and to try to say something useful about what these blockages might mean for reconceiving the ways, means, and ends of education and educational research for and with students outside the mainstream.
This concluding chapter revisits the question of why a human sciences approach to research on learning is necessary and summarizes major themes from across the chapters. The conclusion highlights the need for a democratic practice of educational research, the importance of researchers’ making explicit and participating in the imagination and constitution of new social futures, and the expansion of possibilities for understanding, action, and “liminal” participation in practice as potential teloi of learning.
While much of the recent educational literature has been devoted to explaining how investigators can produce high quality, practical research evidence (e.g., Cook, 2002; Feuer, Towne, & Shavelson, 2002; Shavelson & Towne, 2002; Slavin, 2002; Towne, Wise, & Winters, 2005), little attention has been paid to how evidence can and should be used by teachers and school leaders. Our goal is not to review the empirical literature on teachers’ and school leaders’ use of evidence, but rather to identify the conceptual tools that frame our thinking about this work. Policymakers often work on the assumption that evidence-based practice should be a simple and straightforward process for school practitioners; that is, practitioners need only follow the guidance offered by evidence—typically equated with qualitative research findings and trends in student achievement data—when deciding what they should do and how they should do it. However, this belief is based on several questionable assumptions.
Using the “story constellations” version of narrative inquiry, I tell of two schools—Cochrane Academy and Hardy Academy—that evolved from a shared social narrative history and that were given stories of school and stories of reform that had many features in common.
The use of ethnographic methods yields a rich account of various factors that play a crucial role in determining how educational policy is implemented.
The generic qualities of a qualitative doctoral dissertation proposal are discussed in this article, including how they relate to the dissertation and to the nature of a research university. Standard parts of a proposal are discussed and reasons given for the role each plays.
This article argues for the use of qualitative methods in large-scale evaluations. We demonstrate that despite the challenges it presents, the incorporation of qualitative methods significantly improved an evaluation of the Waterford Early Reading Program and translated into findings that were meaningful and useful to stakeholders.