Evaluative Research Methods: Managing the Complexities of Judgment in the Field

reviewed by Jori N. Hall - July 25, 2017

Evaluative Research Methods: Managing the Complexities of Judgment in the Field positions evaluation in the larger context of inquiry. The author, Saville Kushner, makes the case that all inquiry has an evaluative aspect and therefore requires investigators to make judgments. He further contends public inquiry is linked to civic action and, as a consequence, democracy. It is with this in mind  that Kushners book reignites Democratic Evaluation (DE) and nudges more dynamic relationships among evaluators, program stakeholders and society for improved moral and social change.

Four parts structure the book. Each chapter therein supports the aims and procedures of DE. The reader is advised that the book is not a how-to text per se. Instead, the book theoretically engages democratic principles using real-world examples to support an evaluators decision-making process.

In Part One, Kushner explains the key concepts and intentions of the book. In Chapter One, he clearly defines DE and establishes how evaluative judgments that inform the future of a program are linked to democracy. In Chapter Two, Kushner provides a brief, but thorough historical account of DE as well as an explanation of why it is in need of rejuvenation.

In Chapter Three, Kushner claims argumentation is at the heart of evaluation. Yet, argumentation is moving out of public sectors, which threatens democracy. This issue further substantiates the need to reestablish DE, according to Kushner. He also makes a critical point related to equity that encapsulates his perspective on DE and the complex role of the evaluator:

Democratic evaluation cannot be for an equitable society because there are many definitions of equity, and it is not the democratic evaluators job to insist on one over others. To accept society as it is, but to understand it better and make it more transparent, is a legitimate goal of the evaluator (p. 35).

In Chapter Four, Kushner contends too much attention is given to applying grand theories to settings (testing for correspondence), while insufficient attention is given to practical theorizing (making sense of what is happening as understandings evolve). He also notes it is important for investigators to prioritize the local context when seeking explanations.

In Part Two, Kushner offers methods and case study examples to illustrate DE implementation. In Chapter Five, Kushner outlines how particular methods, such as observation-based interviewing, are uniquely situated to support practical theorizing. He also advocates evaluator self-expression, suggesting that evaluations/programs need to be understood in the context of an individuals (evaluator/stakeholder) lifea central argument made in his previous book, Personalizing Evaluation.

In Chapter Six, Kushner takes issue with the traditional interview typology: structured, semi-structured and unstructured. Opposed to the traditional view, he posits all interviews are structured by factors such as investigator competencies, interview topics discussed, and so on. The interview example provided illustrates how the evaluator can facilitate a rich exchange of information by moving the respondent from describing events to rendering judgments about eventsa method intended to democratize the interviewing process. This chapter works well to demonstrate the distinctive character of democratic-infused interviewing and, as a result, was one of the most informative aspects of the book.

Chapters Seven, Eight, and Nine build on Chapter Six. In Chapter Seven, for example, Kushner explicates how initial observations can be used to inform reflective interviewing and serve as a form of triangulation. In Chapter Eight, Kushner discusses case studies and provides examples. The Missing Persons case study example is helpful to demonstrate how a program is experienced by a variety of stakeholders. And the Bullfighting case study is extremely useful to demonstrate Kushners point about how to make meaning of the evaluation in the context of an individuals life. In Chapter Nine, Kushner places case studies in the context of organizations. Doing so emphasizes the notion that no matter what the case is (i.e., program, person), it is embedded in some sort of organizational structure. Because case studies are considered the study of bounded systems, Kushner sees value in utilizing this design to evaluate organizational systems. From this perspective, case studies enable analysis of democratic aspects within the organization. That is, case studies can facilitate assessments of social behaviors, diverse perspectives, and power struggles.

In Chapter Ten, Kushner proposes the idea that evaluative writing is imbued with democratic aspects because it allows: (a) evaluators to represent others, (b) stakeholders to make judgments about the evaluation, and (c) the public to consume evaluative inquiry. Multiple strategies are given for how to accomplish democratic aims through evaluative writing. Such strategies include: (a) writing a report in negotiation with stakeholders, (b) incorporating poetry, and (c) crafting a writerly text or an evaluative narrative that is open to interpretation (p. 166).

In Part Three, Kushner tackles evaluation challenges and politics. In Chapter Eleven, Kushner uses a narrative from Robert Stake to examine evaluative failure. This narrative captures the idea that failure does not occur in isolation; it is a collective experience. Following Stakes narrative is a failure narrative that reveals the emotional demands of evaluation work. Both narratives serve to bolster a key point discussed in Chapter Eleven: failures in evaluation research need to be examined more closely and more often to understand the intersections of evaluation and democracy.

In Chapter Twelve, Kushner examines the role of the democratic evaluator. Necessary characteristics (i.e., courage, negotiation skills) for conducting DE are outlined and a helpful guide on negotiation tactics is shared. In Chapter Thirteen, Kushner examines how evaluation can engage politicians at the local level. This Chapter is written with Barry Kushner, Councillor for a Local Authority in England. Barrys narrative demonstrates how those in local political positions can implement aspects of DE.

In Chapter Fourteen, Kushner states that methods are informed by values. Therefore, he invites evaluators to critically engage their values. Engaging ones values is important because it provides reflection on the moral basis for decisions made in the evaluation context. Part Three ends with Kushner positioning DE as a form of political action, in that it supports negotiation with stakeholders rather than providing them with the right decision.

In Part Four, Kushner reflects on the assumptions of evaluation practice and offers a perspective on the future of evaluation. In Chapter Fifteen, Kushner critiques the use of a signal theory and advises evaluators to embrace the diversity of theories and values in the evaluation setting. In making this point, he directs attention to a democratic imperative: the principle of inclusivity (p. 246). This principle refers to making varied theories and values visible. Although highlighting different and competing values can lead to messy and complex reporting, doing so upholds the pluralistic intentions of DE.

In Chapter Sixteen Kushner notes that while DE is needed in international contexts, accomplishing DE is significantly complex in some settings. This is primarily due to information apartheid (p. 263), which refers to contexts within which access to information and public dissent is severely limited or restricted (p. 263).

Chapter Seventeen begins with the argument that evaluation is often championed to encourage innovation. With this in mind, Kushner takes a closer look at innovation, first problematizing it and then distinguishing it between improvement and change. Part Four ends with Kushner questioning the assumption that innovation always produces positive outcomes.

As mentioned previously, the theme of the book is DE. What are the implications of a democratic-oriented evaluation for society? What is the role of an evaluator when taking a democratic stance? Questions such as these frame the book. With so much attention given to democracy, it seems the book title would be more reflective of the explicit democratic commitments expressed throughout the text. That said, the book is a wonderful addition to evaluation literature. It not only delivers a solid justification for DE, but also emphasizes how to think democratically. The rich examples and tables illustrating the various ways evaluators can implement democratic practices are a major contribution of the book. Also, the discussion on practical theorizing is an important addition, particularly because the topic of local theorizing is often limited to a discussion of logical models or inadequately addressed in other evaluation texts. Moreover, Kushners writing utilizes the democratic principles advanced in the text. That is, the writing includes strong arguments and welcomes engagement and critique. Because the book provides an in-depth exploration of a particular type of evaluation approach, the text could be used for an evaluation course with students that have a foundational knowledge of evaluation. The book could also be used to supplement an introductory evaluation course that includes democratic-oriented evaluation stances. Ultimately, any evaluator committed to social justice, equity, and democracy could find rejuvenation in those topics from Kushners text.

 

 

Cite This Article as: Teachers College Record, Date Published: July 25, 2017 https://www.tcrecord.org ID Number: 22102, Date Accessed: 11/27/2021 10:05:38 PM Purchase Reprint Rights for this article or review