Scientific Research in Education

Approaching the task of reviewing Scientific Research in Education, I had a faint understanding of the significant challenge faced by the National Research Council Committee charged with conducting a study to “…review and synthesize recent literature on the science and practice of scientific education research and consider how to support high quality science in a federal education research agency” (p. 22).  Offering comment on the policies that might structure the flow of federal research funds in education exposes the Committee to the criticisms of any and all who view their interests as at stake, i.e., every serious educational researcher.  As if providing advice to the federal government in this area were not a sufficiently daunting task, the Committee and the editors of this report took it upon themselves to offer advice to the entire field of education research. 

This book is an act of courage, for which the members of the Committee should be applauded even by those who disagree with the advice presented.  The field of educational research is sprawling and diverse, and it is virtually impossible to craft a statement of principles to guide its development that would meet with the approval of all segments of the profession.  The directness of the approach and the clarity of the advice offered are both long overdue in a profession that receives too much criticism and too little constructive advice.  Encouraged, indeed inspired, by the work of the Committee presented in this volume, I will try to live up to their model of directness in this review.  I will proceed by alternately discussing the main points raised in each chapter and offering a critique, and then I will conclude by raising some more general points for consideration as the Committee and the field work to extend and expand on this discussion.

The book proceeds in six chapters.  The first chapter provides a brief statement on the historical and philosophical context of U.S. education and notes five dimensions of U.S. education that are particularly relevant to the report: 

• the emergence of refined models of human nature;
• progress in understanding how scientific knowledge accumulates;
• recognition that education is a contested field of study;
• new developments in research designs and methods; and
• increased understanding of the nature of scientific rigor or quality (p. 15).

This chapter also presents the charge to the Committee and the following three questions used by the Committee to frame its work:

• What are the principles of scientific quality in education research?
• How can a federal research agency promote and protect scientific quality in the education research it supports?
• How can research-based knowledge in education accumulate? (pp. 22-24)

The chapter concludes with five assumptions guiding the work of the Committee, including:  1) an inclusive, but structured view of the science of education, 2) a recognition that many studies will result in failure, 3) that it is possible for multiple observers of educational phenomena to agree on what they see, 4) that scientific foundations alone will not improve the value of educational research, and 5) that scientific inquiry can contribute to the improvement of education particularly when combined with other approaches to understanding human behavior.

The Committee may have underestimated the impact of the contested nature of education, particularly in shaping attitudes toward and support for education research.  The Committee identifies several reasons why the public may not support education research, among them research quality, fragmentation of the research effort, and the divide between research and the practice of education.  Later the Committee documents the continuing low levels of support for research in education at the federal level.  The Committee also makes it clear that policy makers and educators will draw on values along with science-based education research in determining policies, and that research will be unlikely ever solely to determine policies and practices in education.  If this is so, and I think it is, then the prospect that education research is supported to foster debate and not to resolve issues of policy and practice must at least be considered.  Such an interpretation offers a reasonable explanation of funding levels for education research.  Moreover, if education research is used primarily to provide the garnish for values-driven debates, the prospects for greater support, even for scientifically-based education research would not seem great.

Chapter 2 argues that the accumulation of scientific knowledge in the field of education is possible and that programs of research leading to such cumulative knowledge share certain common characteristics.  To illustrate the possibilities for knowledge accumulation in the field of education the Committee presents four examples:  the study of gene activation in molecular biology; progress in the assessment of human performance, including educational achievement; research on phonological awareness and the development of reading skills; and studies of the impact of educational resources on student outcomes.  From these examples the Committee identifies three enabling conditions for scientific knowledge accumulation – time, fiscal support, and public support – and four characteristics of scientific lines of inquiry – a fits and starts pattern of progress, continuing professional critiques among the scientists involved, the interdependent nature of theory, method and empirical findings, and the inherent complexity of studying humans. 

One might quibble with the choice of examples of lines of research, particularly the work on the impact of resources on learning that has been so bereft of solid theoretical progress and so highly politicized and opportunistic. However, the examples, particularly if interpreted to represent lines of research at different stages of development, do serve to reinforce the Committee’s major claim that cumulative knowledge is both possible and valuable in the field of education.  Less satisfying is the discussion of the conditions that enhance the prospects for knowledge accumulation.  The Committee offers only a cursory explanation in this regard, and notably absent is any mention of the kind of training that education researchers should receive to equip them to plan, execute, and report on studies that might support accumulation.

In Chapter 3 the Committee calls for the strengthening of a scientific community or culture within the field of education research and identifies guiding principles for such a community.  The Committee notes six principles as being essential:

• Pose significant questions that can be investigated empirically
• Link research to relevant theory
• Use methods that permit direct investigation of the question
• Provide coherent, explicit chains of reasoning
• Replicate and generalize across studies
• Disclose research to encourage professional scrutiny

The chapter proceeds to apply these principles to two current genres of scholarship in education – connoisseurship and portraiture – and concludes that neither form of scholarship meets the requirements of scientific inquiry.  The chapter concludes by considering two examples of research the Committee views as meeting the requirements of scientific inquiry, although to different degrees.  The Committee argues that stronger science will result when work is grounded in theory, when it considers multiple competing empirically testable explanations, when it exhibits a more transparent chain of reasoning, when random assignment is employed to address alternative explanations, and when the work is replicated and extended.

There is really nothing very new or startling set forth in this chapter.  The attention to the development of a scientific culture or community of scientists is quite consistent with social science research on science and scientists. The Committee’s major points are well stated and well illustrated.  The choice of well-known examples of educational scholarship that clearly fall outside the Committee’s definition of scientific inquiry is helpful in communicating the boundaries of science in education research.  If anything, the tone of the chapter may be too gentle to capture the attention of the education research profession.   

While the Committee stresses the features of scientific education research shared with all scientific endeavors in Chapter 3, in Chapter 4 it turns to unique aspects of education and of educational research that set the scientific study of education apart from scientific investigation in other fields.  The Committee notes that scientific education research must contend with five features of education, each of which contributes to the overall importance of developing lines of research that carefully consider context:

• The value laden and political nature of education that denies education researchers a single goal around which to orient their work

• The central involvement in education of people, including students, teachers, parents, citizens, and policy makers, who each exercise will or volition and so deny education researchers the level of control enjoyed by investigators of other phenomena

• The variable and rapidly changing nature of educational programs that present researchers with unstable conditions under which to perform their work and draw conclusions.

• The interdependent and multi-layered nature of the organization of the educational system that requires complex analysis strategies

• The diversity of students individually and in subgroups that presents educators and educational researchers with fundamentally different circumstances within formally similar institutions.

The Committee concludes that these features of education make it particularly important for education researchers to consider context in any attempt to generalize from scientific research.

Chapter 4 concludes with a discussion of three key features of education research: its multidisciplinary nature, the ethical challenges posed by fieldwork involving vulnerable populations such as children, and the need for education researchers to rely on practitioners to conduct much of their work. 

Although the Committee looks at the features of education that complicate the work of education researchers and concludes that scientific work will need to consider context in any attempt to generalize, others will conclude that generalization in the scientific sense is impossible under such conditions and so will move away from a scientific approach to research.  This is a difference of opinion that will be difficult to overcome.

Chapter 5 presents the Committee’s ideas on research design.  The Committee notes that there are many legitimate scientific approaches for research in education, that some designs are better than others for addressing particular questions, and that over time different types of inquiries and methods are necessary to generate knowledge in an area.  The chapter divides education research questions into three major types, those dealing with description, cause, and process.  Examples of research addressing each type of question are considered to further illustrate the thinking of the Committee.

The chapter concludes with two major suggestions designed to support improvements in education research.  First, the Committee notes that in many areas of education our basic theoretical understanding is quite weak, and it calls for more attention to descriptive and theory-building studies.  Second, the Committee observes that educational reforms are typically undertaken without sufficient scientific evidence to guide either their development or their implementation and refinement.  To address this issue the Committee calls for more systematic study of the implementation process, particularly in diverse educational settings, and for more research to identify causal relationships, largely based on the expanded use of random assignment with in-depth qualitative studies used to ferret out nuances and additional causal claims.

While there is much that is useful in this chapter, it is telling that the Committee included a chapter on design only, while acknowledging that design is just one aspect of the process of scientific investigation.  The Committee argues that design is often the subject of much debate and that alone seems to justify the chapter. But this emphasis on design is curious in light the of Committee’s own repeated points about the importance of theory and in view of the near absence of sound instruction in theory and theory construction in graduate programs preparing education researchers. 

Equally curious is the prominence of the Committee’s recommendation that the use of randomized field trials be expanded, particularly in light of the Committee’s extensive discussion of the turbulent nature of educational policy and educational organizations that place significant limitations on such fieldwork. The Committee’s view that education research programs could include diverse methods employed in series of studies over time would permit a combination of controlled experiments and field studies linked by well articulated theoretical frameworks to achieve both a better understanding of causation and a more realistic assessment of the operational implications of any findings. 

The Committee makes repeated mention of the potential for qualitative and descriptive research to contribute to theory building, but it offers very little compelling evidence that such methods are typically used to launch new theoretical lines of research.  More attention to models of the use of qualitative research in this way might assist the field in understanding the kinds of contributions the Committee envisions for this type of work.

In Chapter 6 the Committee takes up its specific charge to consider how to support high quality science in a federal education research agency.  The Committee describes its approach to this task as one of creating conditions to foster a scientific culture within a federal education research agency.  It makes an important distinction between its approach of creating an agency that is really a community with the norms and values of science and the approach in recent federal legislation of listing methods that represent a scientific approach.  The Committee argues that using methods to define science ignores the central role of theory and a skeptical, self-regulated community of researchers, both of which are essential to guarantee quality.

The Committee proposes six principles to guide the design of the federal education research agency:

1. Staff the agency with people skilled in science, leadership, and management.
2. Create structures to guide agenda, inform funding decisions, and monitor work.
3. Insulate the agency from inappropriate political interference.
4. Develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to policy and practice.
5. Adequately fund the agency.
6. Invest in research infrastructure.

The chapter provides guidance on the immediate implications of each principle.

In the conclusion to the chapter the Committee observes that its design principles for the federal education research agency are rather intuitive and self-evident, but it also points out that it has been over twenty years since a federal education research agency operated under the basic conditions necessary to form a scientific culture and pursue a mission of scientific investigation.

Overall, this volume offers a well-considered view of the potential for scientific research to advance educational practice.  The advice is sensible and moderate in both substance and tone, and, as such, it should serve to engage a wide-range of education researchers in thoughtful reflection on the current state of the field and on important questions regarding its future.  The Committee itself notes the continuity with earlier reports of the same nature.  There are a few issues that are sure to raise concerns from some education researchers, but the Committee has done its best to craft a document that will unite the majority of those interested in scientific research in education to strengthen the scientific culture referred to throughout the book. 

In keeping with the Committee’s call for a skeptical community of education researchers, I want to note a few general issues that might be worth more attention as the discussion of scientific research in education and the role of the government moves forward. 

First, the Committee understandably said very little about non-scientific education scholarship beyond illustrating the boundary between scientific and non-scientific work.  A discussion of the value of non-scientific scholarship is in order along with a discussion of the role the federal government might play in fostering such work.  If the major federal education research agency develops the kind of scientific culture advanced by the Committee, it is not likely to be a home for scholarship that is not scientific in orientation.  It is possible for other agencies of the federal government to support such work, but it is also possible that it will be neglected, at least by the government.  Since non-scientific scholarship can be an important source of the theoretical insights that the Committee views as essential for the progress of science, short-changing it in the overall federal investment might impede the kind of progress the Committee envisions.  Of course, the Committee might reasonably conclude that non-scientific work is best supported in other ways.

Second, despite repeated calls for a balance of empirical and theoretical work to support the accumulation of knowledge, the Committee’s report emphasizes empirical work to the near exclusion of serious discussion of theoretical work.  The inclusion of a chapter on design without accompanying chapters on the other major aspects of scientific work is the most obvious evidence of a disproportionate focus on the empirical aspects of research.  The report might have also included chapters on question formation since so many of the questions under girding current education research do not permit scientific work.  The Committee uses the debates on design as a rationale for including an entire chapter devoted to design, but at a time when many education researchers do not distinguish vague notions from formal theories, when many do not appreciate the difference between a concept and a measure, and when hypotheses, when they appear at all, in reports on education research are seldom truly testable, there is ample reason for the Committee to take seriously its own advice on the importance of theory and provide much more guidance for both the field and the federal agency it envisions.  The Committee seems to appreciate the rather dramatic imbalance between solid ideas and available data in the field of education research, and it might have considered how to encourage more careful disciplined thought and less unguided data gathering, a good bit of which is currently supported by government resources.  The Committee’s discussion of the importance of common variables and datasets without corresponding attention to the development of common conceptual tools is another example of the empiricist bent. 

Third, although the Committee’s charge directed it to consider the conditions for creating a federal agency devoted to scientific education research, it might also consider the conditions in universities and other research organizations in which most education research is currently conducted.  There may be certain features of these working environments for researchers that are less receptive to scientific research than might be ideal.  For example, years of inadequate funding from all sources have left many institutions where education research is done without the appropriate infrastructure to support such work.  One result of this is to place a substantial burden on individual investigators and individual studies to support the costs of inquiry.  With costs escalating in general and inadequate institutional supports, too many investigators may tend to shy away from the more ambitious research envisioned by the Committee.

Such infrastructure deficiencies include not only facilities and equipment, but also trained personnel and institutional cultures consonant with the requirements of scientific work.  To take but one example, the Committee correctly assumes at the outset that many scientific studies in education will result in failure, but there is not widespread appreciation of this fact among dissertation committees, tenure committees, and journal reviewers in education.  Work will need to be done to change such expectations if we want to encourage investigators at all stages of career development to pose hypotheses than can actually be tested.       

Fourth, the tone of the Committee’s report properly conveys the challenges of conducting sound scientific work, and the report calls attention to the canons of scientific design.  Moreover, because the Committee’s charge came from the federal government, the entire enterprise has an establishment caste.  This is likely to be viewed as something less than a congenial approach by those researchers who view their role as challenging the status quo and opening up new possibilities for education.  But the Committee might also highlight the potentially revolutionary nature of scientific education research.  Some of the most profound and sustained revolutions in thought over the past five hundred years have begun with scientific research, and there is no reason to believe that educational thought can escape the same fate in the face of an energetic, robust, and persistent scientific education research effort. 

Finally, the Committee’s report lays the groundwork for conversation that could lead to the revitalization of the field of education research at a time when many would agree such revitalization is needed.  For this to happen both the work of the Committee and the conversation in the field will need to continue.  The Committee’s report provides a fine point of departure.