The Last Word in Research on Teaching - A Review of the Handbook of Research on Teaching, 4th Edition - Part 2 - Methodology
reviewed by Christopher M. Clark, Mark Girod, Ebony Roberts, Jaime Galindo, Patricia Aben, Sean Farmer & Lanetia Noble - 2003
Title: The Last Word in Research on Teaching - A Review of the Handbook of Research on Teaching, 4th Edition - Part 2 - Methodology
Author(s): Virginia Richardson (ed.)
Publisher: American Educational Research Association, Washington
ISBN: 0935302263, Pages: 1278, Year: 2001
Search for book at Amazon.com
OVERVIEW BY MARK GIROD
The tension between old and new framed in Foundations carries into Methodology. The continuing tension is most apparent in the bookends of this section: Crawford and Impara's Critical issues, current trends, and possible futures in quantitative methods, and Zeichner and Noffke's Practitioner research. Crawford and Impara scan the field broadly, ultimately suggesting that researchers should be conducting more experimental studies because teachers and administrators want and need definitive findings useful for making difficult high-stakes decisions. Zeichner and Noffke, in a call for new methods, describe the burgeoning field of action research. They recognize the concern for rigor in practitioner research but argue eloquently for a re-definition of validity. Their provocative reconceptualization of the epistemology of research on teaching makes room for more attention to local and contextual validity, as well as to the voices of teachers in the councils of research on teaching. In a powerful essay Donmoyer (Chapter 11) declares that the paradigm wars are over, or at least should be left to die. He suggests that this is the wrong battle, one that leads us into inefficient and unproductive tradition bashing.
One of the most vigorously new contributions in the Handbook is the seven-chapter set on "Special topics in qualitative methodology." Virginia Richardson and her advisors clearly must have decided that the diversity of issues and tools in the qualitative tradition could not adequately be represented by a single chapter. Certainly choosing to present a more variegated representation of qualitative methods than of the quantitative suggests where to look for the most active current developments and contending methodological values in the field of research on teaching.
Howe examines the qualitative tradition through the eyes of a philosopher. Equally powerful, and even more radical, is Lather's chapter on the inappropriateness of current views of validity. Like Zeichner and Noffke, Lather issues a call for radically new definitions of "goodness" in research. These two chapters may well become widely cited classics by the next generation of researchers on teaching. This section is rounded out by Eisenhart's re-conceptualization of culture and what it means to do ethnography, Gudmundsdottir's narrative research and writing, and Jennifer Greene's suggestions for combining quantitative and qualitative methods. Each has something different and new to offer.
Chapter seventeen by Porter, Youngs, and Odden examines current practices in assessment of teachers and teaching. In an effort to narrow their task, these authors report the histories and latest developments in three teacher assessment programs: ETS and the NTE/Praxis tests, NBPTS, and INTASC. Unfortunately, this chapter doesn't fit well with the others in the Methodology section. Rather than push readers to think differently about inquiry, this chapter bogs down in programmatic details.
Together, the chapters on methodology provide an interesting but occasionally confusing mix of old and new. Arguably, more progressive thought has taken place in the qualitative tradition, and the Methodology section of the Handbook reflects this changing intellectual climate.
CHapter 10. Critical Issues, Current Trends and Possible Futures in Quantitative Methods
Crawford, J. & Impara, J.C. (2001). Critical issues, current trends and possible futures in quantitative methods. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 133-173).
Review by Jaime Galindo
Crawford and Impara review recent developments in quantitative methods in research on teaching. They begin by discussing the need for between-group studies that compare the use or application of a method or idea in one trial to a control group for the purposes of determining the effect of that method or idea. The authors go on to discuss the use of meta-analyses as another means of providing answers to important questions by systematically drawing data from multiple studies. Both methods are useful, but Crawford and Impara go on to claim that experimental between-group studies are the standard toward which to strive while meta-analyses should be used for confirmation rather than decision making.
Believing that education lacks a sufficient quantity of large-scale true experiments, Crawford and Impara cite the Tennessee Study known as Project STAR conducted from 1985-1991 as an example of important research providing solid empirical evidence (in this case, the value of reduced class size on K-3 student learning). It is a between-group study that, while expensive, shows a consistent impact of a particular treatment that educators and school boards can use.
A specific area that the authors highlight for review is that of performance assessments, or measures of student achievement. Crawford and Impara believe that insufficient attention is being given to the dependability of current assessments. Tools that have such a “high stakes” impact on student lives need to be reconsidered in light of the unreliability of what they purport to tell. Discussion ensues on the use of generalizability theory for estimating dependability.
The second part and largest portion of the chapter deals with the quality of the measures used in research studies. The authors feel that the quality of the measures must be determined independent of the research question and that the choice of appropriate measures should be a prime consideration in both design and analysis. Threats to internal validity are virtually impossible to eliminate and may be associated with many aspects of a measure (e.g. testing, instrumentation, regression, and selection interactions). Measures typically produce unstable scores and increase the error in estimating treatment impact. The authors suggest ways to reduce the impact of instability, but such solutions are not always possible. They discuss threats to construct validity that include inadequate operational definition of constructs, bias due to the measurement or presentation of a variable in a limited form, subjects’ and experimenters’ response to unintentional aspects of the experiment, differential impact of a treatment at different levels, restricted generalizability due to too specific a measure, and effects of the order in which multiple measures are applied.
One aspect of between-group studies that the authors detail is the need to divide subjects into two or more groups. Cut scores are specified to divide one group from another. The authors feel that in most cases, greater power and robustness are achieved by defining groups of equal size using normative methods. When the desire is to group based on specific attributes, assignment is typically achieved using methods that are criterion-referenced, empirical, examinee-centered, or test-centered.
As to the specific choice of measures, Crawford and Impara provide advice on locating existing measures, available both commercially and non-commercially. In addition, the authors conducted a survey of researchers in the Teaching and Teacher Education division of the American Educational Research Association to draw conclusions about where the field is currently positioned and where research on teaching may be headed. They also summarize recent research by M. Kennedy that asked teachers what types of research they find persuasive, relevant, and influential. Comparing the two, and adding additional commentary from former AERA presidents, the authors conclude that priorities need to be redefined to produce more research that delivers what educators and school boards need to make informed decisions to improve educational practice.
While Crawford and Impara aim “at providing an update regarding quantitative methods for the field of research on teaching” (p. 158), they appear to draw most of what they say about measurement from Campbell and Stanley (1963) and Cook and
Chapter 11. Paradigm Talk Reconsidered
Donmoyer, R. (2001). Paradigm talk reconsidered. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 174-200).
Review by Mark Girod
Donmoyer critiques Kuhnian inspired "incommensurable paradigm talk" with four arguments appealing to issues of epistemology (talk of verisimilitude), changing times (acceptance of some level of constructivism), need for theoretical pluralism (and problems of doing so) and the problem of "balkinization" (de-valuing and even attacking paradigms different from our own). He continues with a similar critique of Shulman (1986) and Gage's (1989) "complimentary paradigm talk" which argued that new paradigms were simply adding to the dominant (positivistic/process-product) paradigm. Donmoyer deconstructs the "complementary paradigms" argument with a discussion of the vastly different notions of validity, reliability, theory, theorizing, and politics to which each subscribe.
Donmoyer suggests that wasting further time, energy, and journal space in paradigm talk is unproductive because the logic of these arguments is fallible. Instead, he suggests we move from comparing perspectives on issues of epistemology to issues of purpose. He offers a five-category framework: "(1) the "truth" seeking purpose; (2) the thick description/local knowledge purpose; (3) the developmental, quasi-historical purpose; (4) the personal essay purpose, and; (5) the praxis/social change purpose" (p. 190). Donmoyer suggests this more productive framework "will help us talk with and learn from each other about the complex processes of teaching and learning and that such talk will result in more thoughtful teaching and more balanced educational policymaking" (p. 195).
I found Donmoyer's argument and the historical retrospective he uses to build it interesting and engaging. His point is well taken--unproductive perspective-bashing leads us away from progress and away from thoughtful research in general. However, he cautions and I agree that his "purposes" categorization may not be the solution to unproductive perspective talk either.
Chapter 12. Qualitative Educational Research: The Philosophical Issues
Howe, K.R. (2001). Qualitative educational research: The philosophical issues. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 201-208).
Review by Mark Girod
Howe examines qualitative (interpretivist) educational research through four philosophical lenses: epistemology, ontology, political theory, and ethics. Howe splits the interpretivists into two camps based on their answers to issues of constructivist epistemology. The first camp, the postmodernists, believe "knowledge is merely a cultural-historical artifact," that it is "merely a collection of moral and political values," and that it "serves certain interests and purposes" (p. 202). The second camp, the transformationists, agree in rejecting the philosophical quest for ultimate truths but "see their task as working out defensible conceptions of knowledge and rationality that have contingent human experience as their basis" (p. 202). Among the transformationists, Howe includes pragmatists, critical theorists, and certain feminist researchers.
His categorization scheme seems useful for the purposes of his discussion although not all would agree with his distinctions. For example, Howe implies that the postmodern commitment to situatedness inevitably leads to nihilism. He suggests, therefore, that anyone making a real contribution to education and society is "crossing the line" and is actually a transformationist. Howe concludes,
In the wake of the interpretive turn, the philosophical debate is now between those who seek some new understandings of knowledge, rationality, truth, and objectivity (i.e. transformationists) and those who are ready to abandon these concepts as hopelessly wedded to the bankrupt modernist project (i.e. postmodernists) (p. 207).
In the end, Howe's discussion is productive as it shows interesting similarities and differences between camps within the interpretivist framework. His discussion of political theory and ethics in the interpretivist camp is most enjoyable and I appreciate his commitment to expression of his own sometimes radical ideas.
Chapter 13. Changing Conceptions of Culture and Ethnographic Methodology: Recent Thematic Shifts and Their Implications for Research on Teaching
Eisenhart, M. (2001). Changing conceptions of culture and ethnographic methodology: Recent thematic shifts and their implications for research on teaching. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 209-225).
Review by Mark Girod
Eisenhart divides her discussion into two parts. In the first, she examines shifting conceptions of culture in anthropology and sociology, offering suggestions for future research on teaching. She outlines traditionally held views of culture and research on culture then contrasts these with more current definitions and perspectives such as those used by postmodernists, ethnic, feminist, and homosexual researchers. Eisenhart suggests a shift in focus on cultural productions--attention to cultural forms, physical and symbolic representations, and the organization of people and systems across time and space. These shifts foreshadow changes in methodology because "such issues are not the ones that have drawn the interest of most people who are presently engaged in discussions about improving ethnographic techniques" (p. 218).
In the second half of her chapter Eisenhart suggests ethnographers have not been daring enough to alter their methodology to reflect new definitions of culture. Changing cultural definitions have led to "more collaborative models of the relationship between researcher and researched and to experiments in writing (so-called "textualist strategies") that allow more different perspectives or voices to be revealed in final accounts" (p. 219). Eisenhart suggests new methodological designs employing multi-site and multi-timed ethnographies of events, movements, and interactions, as well as groups.
I enjoyed the broad range of examples Eisenhart used in making her argument and look forward to research inspired by her advice.
Chapter 14. Narrative Research on School Practice
Gudmundsdóttir, S. (2001). Narrative research on school practice. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 226-240).
Review by Mark Girod
This chapter is organized into three distinct sections, each with separate footnotes. Gudmundsdóttir provides some historical background in the first section explaining that "student learning, as a unit of analysis, disappeared with the fading away of the process-product research tradition" (p. 237). She argues that school practice is fundamentally socially and culturally mediated making cultural-historical activity theory (CHAT) the appropriate set of ideas to frame narrative inquiry research. She concludes that narrative research, because of it's affiliation with CHAT, is the timely and appropriate tool for exploring the complexities of school practice.
In the second section Gudmundsdóttir further articulates her views on narrative inquiry methods. She describes successful narrative inquiry as "this 'trafficking' between the parts (individual episodes - first level) and the whole (theory or the narrative - second level) and how they give meaning to each other that constitutes narrative interpretation" (p. 230). The section ends with a discussion of narrative as cultural "scaffolds" or "scripts" that help us "make sense and give us the opportunity to climb to higher ground" (p. 232) as well as some examples of teachers using narrativized curricula.
The third section includes a discussion of "the language of practice" arguing that "it is through the language of practice that social organizations and relationships are defined and tried out, and the language of practice becomes the place where our identity as teachers and researchers, our subjectivity, is conceived" (p. 234). Gudmundsdóttir ends the chapter with a discussion of appropriate "voice" in research reports, ethical issues involved in narrative inquiry, and a short concluding section that further pushes narrative inquiry as a useful tool in research on school practice.
I found the unusual structure of the chapter a bit distracting. The ideas within were interesting and compelling, yet it was difficult for me to follow the ideas within the conversation. Overall, the chapter is enjoyable to read and I found myself thinking about it several days later–a mark of any provocative text.
Chapter 15. Validity as an Incitement to Discourse: Qualitative Research and the Crisis of Legitimation
Lather, P. (2001). Validity as an incitement to discourse: Qualitative research and the crisis of legitimation. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 241-250).
Review by Mark Girod
Lather discusses shifting epistemologies and resulting implications for the notion of validity in educational inquiry. Although not meant to be a lesson on the history of validity, Lather takes the time to explore its genesis and important developmental milestones. She states, "the major function of the chapter is to posit both the intelligibility and availability of alternative discourse-practices of validity" (p. 242).
Lather explores validity from both the perspective of positivism and post-modern, post-positivism. The discussion moves from validity as "objective correspondence with reality" to validity as something "situated" and "perspectival." An interesting insight is Lather's discussion of validity as having "power, the power to determine the demarcation between science and not-science" (p. 243). The desired endpoint is for validity to be viewed flexibly allowing us "to use validity to further change the terms of the legitimation of knowledge beyond discrete methods and toward the social uses of the knowledge we construct" (p. 247-248).
Lather gives great attention to the evolution of Guba and
Chapter 16. Mixing Social Inquiry Methodologies
Greene, J.C. (2001). Mixing social inquiry methodologies. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 251-258).
Review by Mark Girod
Greene argues with fervor that disparate methodologies, epistemologies, and inquiry frameworks can and should be combined "to probe more deeply and stretch in new directions the boundaries of what is known" (p. 252). She argues that "the proliferation of divergent inquiry frameworks within science reflects the contemporary raced, classed, gendered differentiation of the broader society" (p. 251). From this, it follows that "the primary rationale shaping the mixed-method commitment to honor difference is the desire to understand more fully, to generate insights that are deeper and broader, to develop important knowledge claims that respect a wider range of interests and perspectives" (p. 251).
Greene offers five different mixed-method designs, each with a different purpose: (1) triangulation designs attempt to increase validity; (2) complementary designs measure "overlapping but distinct facets of the phenomenon;" (3) development designs inform further research; (4) expansion designs mix methods to extend the "breadth and range of the inquiry;" and, (5) initiation designs juxtapose findings in search of unique questions or paradoxes.
The last third of the chapter is devoted to "how-to's" on mixing methodologies in which Greene writes with passion and idealism. She divides the options into combined methodologies that are coordinated in efforts, assisting or complementing one another, and integrated methodologies that combine efforts to achieve something more or beyond what could have singularly been accomplished. She also offers a short caution describing when not to mix methods. Greene's ideas are idealistic and inviting yet I was left wanting more practical advice.
CHAPTER 17. ADVANCES IN TEACHER ASSESSMENTS AND THEIR USES
Porter, A.C., Youngs, P. & Odden, A. (2001). Advances in teacher assessments and their uses. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 259-297).
Review by Mark Girod
This chapter consists of seven major sections each loaded with information about recent work on the assessment of teachers. The sections are: (1) a discussion of the purposes of teacher assessment and a review of previous work in this area focusing on teacher effects research and the emergence of pedagogical content knowledge as a construct of critical importance; (2) new approaches to teacher assessment as developed by the National Board for Professional Teaching Standards (NBPTS), the Educational Testing Service (ETS), and the Interstate New Teacher Assessment and Support Consortium (INTASC), including cost analyses for large-scale implementation; (3) psychometric analyses of these three assessment systems with an emphasis on validity, reliability and fairness, including testing bias and equity issues; (4) innovative teacher assessment systems in Connecticut, California, Cincinnati, and Rochester designed to mentor beginning teachers and support struggling teachers; (5) attempts to assess teachers in terms of student outcomes focusing on the Tennessee Value-Added Assessment System; (6) connections between innovative teacher assessment systems and compensation systems including new salary schedules, and; (7) summary and suggestions for further research including an interesting discussion about the kinds of assumptions most assessment systems make about the purposes of schools, teachers, and education.
This chapter is rich with highly specific information about three national teacher assessment programs. The authors are careful not to advocate for these programs, only offering warranted criticism and praise. In this sense, the authors were good reporters. The chapter could have described more thoroughly the emerging work emphasizing student learning gains as a measure of teacher effectiveness. Innovations of this kind are being developed in
Chapter 18. Practitioner Research
Zeichner, K.M. & Noffke, S.E. (2001). Practitioner research. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 298-332).
Review by Pat Aben
With the current shift in attitudes, ranging from viewing teachers as merely “consumers of educational research” to viewing them as “producers of educational research," the authors address the issue of the “trustworthiness of claims” in practitioner research, as well as specific aspects of practitioner research involving forms of educational inquiry. Zeichner and Noffke lead the reader through a history of similarities and differences between various traditions associated with practitioner research in
The authors contend that since practitioner research involves the study of participants in educational practices, it allows for a better understanding of teacher and student experiences (as well as what and how they learn). While it is a different form of research, it does not often occur in isolation but is done in collaboration with children, other practitioners and researchers, and in many cases is…“from” and “with” rather than “on” with full participation by those affected by the research process. They cite examples of criteria used for assessing the internal and external validity of practitioner research (or trustworthiness) and call for different methods of assessment in order to judge the quality of research done by K-12 educator-researchers.
I found this chapter to be a valuable source of information. Zeichner and Noffke present both positive and negative viewpoints in discussing past and present attitudes toward practitioner research. They observe that for teachers to combine teaching and research into a compatible/acceptable field of educational inquiry has been criticized by researchers as being inferior, especially since teachers are not generally trained in proper research methods. Unfortunately it has been my observation that many teachers view educational researchers (and their research) with skepticism, since many researchers have no practical K-12 classroom experience. Since it is university researchers (not practitioners) who set the criteria for “good” research, as a teacher researcher I hope that the call for an understanding of different methods of assessment does not reduce the expectations for the quality of such research, and that it enhances collaboration between researchers and practitioners.