For The Record: On the Role and Importance of Reviewers
by Gary Natriello - 1996
The production of scholarly journals depends on the willing assistance of those who review the papers submitted for publication. Although many scholars participate in the review process and published authors certainly know of the influence of reviewers, many younger scholars are uncertain of the role of reviewers and unsure of themselves when placed in that role. Even more experienced scholars differ substantially in the time and care they devote to the manuscripts they receive to review. I want to take this opportunity to discuss the role of reviewers and the importance of participating in the reviewing process.
The process of blind peer review such as the one used at the Teachers College Record offers a number of advantages to scholars and the fields in which they work. First, because the identities of authors are kept confidential, authors can be sure that their work will be judged on its own merits with no attention to their prior reputations. This is particularly important for younger scholars who must compete with colleagues many years more experienced. Work by younger scholars receives the same attention from reviewers as the work of established names in the field. But evaluating papers on their own merits also carries benefits for experienced scholars because their work receives careful scrutiny that can save them from making mistakes that might be overlooked when their work is read by individuals impressed with their earlier efforts. Of course, the second benefit of the review process is that because the identities of the reviewers are held in confidence by the editors, reviewers can express their criticisms without reservations. In general this allows reviewers the opportunity to communicate their genuine reactions to a paper. As we all know, sometimes the most difficult thing to get from close colleagues is an honest appraisal of our new work.
There are disadvantages to the blind review process as well. Authors may not understand the perspectives of reviewers whose identities they do not know and so may not know quite how to interpret some of the criticisms their papers receive. Reviewers whose identity is protected may feel free to engage in vituperative rhetoric while expressing their criticisms. In both cases editors can mitigate these negative aspects of the blind review process. Editors can provide authors with general information about the perspective or school of thought in which a particular reviewer is rooted. Editors can also take steps to temper the most caustic comments of reviewers. This second situation presents editors with dilemmas in which the correct course of action is not always clear. At a recent meeting of editors the discussion turned to strategies for handling particularly nasty comments from reviewers. Some editors felt that authors were always entitled to the complete unedited comments of reviewers, no matter how outlandish. Others argued that editors should excise the most scathing comments if they did not contribute to a more complete understanding of the flaws in the paper.
In my first year editing the Teachers College Record, I had at least one occasion when I had to decide how to interpret harsh reviewer comments for an author. I found myself unable to eliminate the offending comments and leave much of the original review so I chose to send a rather nondescript letter to the author without the comments of the reviewer. When the author asked for the reviewer comments, I sent the unedited comments. Later I discovered that the author was quite distressed at the comments. I do not believe that there is a single correct procedure in these cases. Although the review process would be stronger if reviewers refrained from offensive comments, authors may learn of the vehemence of the reaction to their work from at least some quarters by receiving those comments. Clearly the preferred reviewers are those who can comment on a paper in a manner that will benefit authors and scholarship in general.
The selection of reviewers is one way to ensure the quality of the comments that authors receive. Editors use a variety of strategies to identify qualified reviewers. Editors often begin by calling on scholars whose work is well known to them. Although this technique can be useful for identifying individuals who do high-quality work, it does not guarantee the identification of individuals who do high-quality reviews. Sometimes the most prominent individuals in a field are so busy that they are unwilling to take on the task of reviewing, although I am constantly pleased at the large number of very prominent educational researchers who write very substantial reviews of the papers I send them. Editors often identify reviewers by examining the works that are cited in the bibliography of a paper. This process enables editors to designate reviewers whose work is seen as valuable (and sometimes as limited) by the author submitting the paper. Authors should understand that when they cite work in a paper they may also be suggesting reviewers for the paper. Editors may consult directories of professional societies or the programs of recent professional meetings to determine who might have interests similar to those represented in the paper under review. Editors have at least two reasons to match the topic of the paper and the expertise of reviewers. First, such a match generally leads to higher quality reviews. Second, scholars are more likely to take on the reviewing task if the paper under review is related to their own work. Beyond the selection of individual reviewers, editors may select a set of reviewers for a paper so that they offer not only relevant expertise, but also different perspectives. In the case of a general journal such as the Teachers College Record, reviewers may be selected to represent both readers who are specialists in an area and more generalist readers. This allows the editor to judge the broader interest in a paper and whether it communicates to a diverse readership. Of course, when reviewers are selected for different reasons, they may not agree on the merits of a paper.
Younger scholars often ask how they can become involved in the review process. Since editors call on those whose work is known or who participate in professional meetings, individuals who have yet to establish a body of work may not be asked to serve as reviewers. However, I have often received high-quality reviews from younger scholars and graduate students, and I certainly encourage their participation in the review process. Scholars further along in their careers who are not as active in publishing their own work also often make very substantial contributions as reviewers. I want to invite individuals who have not served as reviewers for the Teachers College Record and who wish to become involved to let me know. A brief paragraph describing interests and areas of expertise sent to the editorial office by mail, fax, or e-mail will enable us to include you in our list of potential reviewers. This will most likely result in an invitation to review a paper. Of course, the best way to be sure to be asked a second time is to write a careful and thoughtful review the first time. What makes a good review? Although reviewing the work of others is a major part of the academic enterprise, we typically receive no preparation in just how to prepare a quality review. Although different kinds of scholarship and different kinds of journals suggest special approaches to reviewing, there are some common features that make reviews valuable to an editor and to the author of the paper. First, it is very helpful to begin the review with a short statement detailing the reviewer's understanding of the paper. This is often just a brief descriptive paragraph. Sometimes the reviewer attempts to locate the present paper in a broader body of scholarship. As an editor, I find this kind of statement extremely helpful. It allows me to check the reviewer's understanding of the paper against my own and against the understanding of other reviewers. If there is disagreement about the overall intent of the paper, it may indicate the need to clarify the presentation of the paper or it may indicate that one or more readers have misinterpreted the paper.
Second, some consideration of the conceptual approach taken by the author of the paper is usually in order. What are the major ideas that drive the paper? How are they put together? What is the basic logic of the presentation? Papers differ substantially in their theoretical import, but all papers should attempt to present some essential ideas.
Third, it is important to comment on the technical quality of the paper. Technical quality means radically different things in different traditions, but in terms of the tradition adopted by the author, it is important for the reviewer to assess the adequacy of the effort. Reviewers sometimes also comment on the adequacy and appropriateness of the perspective the author selected for addressing the problem. Such comments are particularly helpful to editors who receive papers from a wide array of traditions.
It is sometimes necessary for reviewers to remark on the organization of the paper and the quality of the writing. A paper that communicates well requires little comment, but one in which presentation is impeded by the quality of the writing should be identified as such by the reviewer.
Reviewers should offer some opinion as to the importance of the paper for advancing a field of scholarship. During times when journals receive many technically proficient papers, ultimate decisions about publication may be made on the basis of import. It is helpful for editors to know whether reviewers think that the paper is important for others to read.
Reviewers should comment on the appropriateness of the paper for the particular journal. Moreover, if a reviewer concludes that a paper is not appropriate for the journal that has it under review but that it might be suitable for some other journal, it is very helpful if the reviewer can identify the other journal or journals.
Finally, reviewers have a dual role; they serve both as gatekeepers and as teachers. In their gatekeeping role reviewers make recommendations regarding decisions to publish particular papers. Reviewers are generally given several options and asked to select a course of action to recommend to the editor. In their teaching role reviewers offer suggestions for ways to improve papers. Often the real strength of a review lies in the quality and thoroughness of the suggestions for improving the paper that a reviewer can provide for an author. The papers submitted to major journals usually involve countless hours of time invested by the author. Reviewers who can identify strategies for authors and the field to realize the benefits of those investments make important contributions to both.