For The Record: Masked Marvels: Anonymous Peer Reviews
by Gary Natriello - 1997
Perhaps no aspect of the academic journal paper submission and review process is more mysterious to beginning scholars than the practice of anonymous or masked peer review. Indeed, many young paper authors are unaware of the peer review process or uninformed about the review practices of particular journals. I have previously written in this space about the review process in general; here I will focus more specifically on the issue of the anonymity of the author and the reviewers. I will deal in turn with the mechanics of the anonymous review process, the advantages and disadvantages of the process, and alternatives for reviewing manuscripts prior to publication.
Mechanics of the Masked Review Process
The masked review process relies on the cooperation of both the author submitting a paper and the reviewers asked to evaluate that paper. By submitting a paper and agreeing to review a paper the parties agree to participate while concealing their identities. The review process is, of course, mediated by the editor and the journal office staff, but the primary responsibility for keeping identities hidden rests with the participants.
The initial responsibility is the author’s. Authors who submit papers to anonymous peer reviewed journals are obligated to follow the appropriate procedures for masking their identities in their written work. There are at least four instances where authors should be concerned about revealing their identities. The simplest instance involves basic author identifying information, often just a name and institutional affiliation. The Publication Manual of the American Psychological Association (1994, p. 248) advises that such information along with other basic author information should be placed on the title page of the paper so that it can be removed by the editor prior to sending the paper to reviewers. Occasionally, I have received papers in which authors have used their names in the running heads to appear on each page of the manuscript. Obviously, this practice makes it impossible to mask the paper for reviewers. Running heads can contain a few words from the title of the paper without revealing the author’s identity.
A second case of information that might reveal the identity of an author occurs when an author includes a citation to his or her own prior work within a manuscript and a corresponding reference in the reference list. Including such materials may present no problem if the accompanying text does not refer to the materials in question as the product of the present author. If I write that "Natriello (1996) pointed to the importance of evaluation processes in determining student engagement with schooling . . . " I may not be revealing my identity. However, large numbers of references to the works of a single author may suggest that the author of the present manuscript and the oft-cited author are one and the same.
A more serious problem arises when the text accompanying a citation identifies the materials as prior products of the author of the present manuscript. If I write that "As I have noted previously (Natriello, 1996), evaluation processes are important in determining student engagement with schooling . . ." there is little doubt that I am the author of the manuscript at hand. In such cases it is essential for the author to substitute either "Author Reference #" or "Identifying Reference #" for his or her name and to place the corresponding reference on the title page so that it may be removed by the editor prior to sending the manuscript out for review. It is not sufficient to mask the name in the citation while including the reference without the author’s name in the regular reference list because it is often quite easy to determine the name of an author from the title and source of a publication.
A third possibility for compromising the identity of an author exists when an author describes his or her role in an event as an integral part of the presentation. An author who played a major policymaking role or a major staff role in implementing a new program will find it very difficult to disguise his or her identity if the perspective afforded the encumbant of the role is central to the argument of the paper and if the individual in the role is widely known. It is the responsibility of the author to alert the editor to such situations when submitting the paper. In such cases the editor may elect to proceed with reviews of the unmasked paper.
A fourth case in which the identity of the author is difficult to conceal happens when an author is so clearly and uniquely identified with a particular type of scholarship that there is little doubt as to the author of a manuscript, or at least reviewers believe there is little doubt. In this situation reviewers are as likely to attribute authorship incorrectly as they are to make a correct identification. Often students or junior colleagues have produced a paper in the tradition of a well-known scholar that reviewers credit to the renowned individual.
Editor’s and Office Staff Responsibilities
Once a paper has been submitted, the editor and the editorial office staff take a number of steps to attempt to ensure the anonymity of those involved in the review process. First, office staff typically review a manuscript for identifying information not appropriately disguised by the author. Office staff can generally discover the most obvious author errors such as citations to the author’s own work that have not been disguised. However, the responsibility for masking an author’s identity remains primarily with the author since office staff can receive hundreds of manuscripts each year and occasionally will miss key pieces of identifying information imbedded in a manuscript.
Second, editors commonly avoid sending the manuscript to reviewers who might know the paper or the author. For example, scholars in the same department or on the same campus, scholars who appeared on the same conference panel where an earlier version of the manuscript was presented, or scholars involved with the manuscript author as mentors or co-authors of earlier work might be avoided to preserve the anonymity of the review process. However, in certain areas where there are only a small number of individuals who are knowledgeable about a topic or method presented in the paper, editors may find it necessary to send the paper to reviewers who know the author of the manuscript. I have found it necessary to do this myself on occasion and have found the comments of such reviewers very valuable, although I always balance such reviews against those prepared by others less closely connected to the author.
Despite the best efforts of editors, reviewers will occasionally receive papers whose authors are well known to them. Reviewers may have heard the paper at a conference or seminar, or may have a relationship to the author of the paper that is unknown to the editor. In such cases reviewers often complete the review and inform the editor that they know the identity of the author when submitting the review. As an editor I find this approach most helpful because it allows me to consider the reviewer comments in the appropriate context and it often minimizes any delay in processing the manuscript since there are generally other reviewers who do not have knowledge of the author’s identity.
Reviewers also have a responsibility to conceal their own identity throughout the review process. The anonymity of the review process extends to the reviewers as well as to authors. Occasionally, reviewers will offer to reveal their identity. Such offers are properly made to an editor in separate correspondence, not in comments to be forwarded to authors. Although there may be rare instances where allowing an author to know the name of a reviewer may be useful, such revelations typically serve no purpose and can unduly influence (positively or negatively) the weight an author might give to a particular set of reviewer comments.
If authors, editors, and reviewers fulfill their responsibilities to the masked review process, anonymity should be preserved. I am often asked if this is really the case. Are identities truly masked? Although it is difficult to know in every instance, I can say that authors and reviewers often believe they know the identity of others involved in the case of a particular manuscript, and when they communicate their suspicions to me as editor, they are just as frequently incorrect as they are correct.
Advantages, Disadvantages, and Alternatives
The masked review process requires care and attention from all concerned. The benefits to be derived from this additional effort have long been recognized. For authors, masked review provides a level playing field in which the current work of well-established figures and newcomers to a field is judged without reference to prior accomplishments. Neither the veteran nor the novice should receive fawning praise or irrelevant criticism based on reputation or the lack of one.
For reviewers, masked review allows for direct comment without fear of retribution or hope of recompense. Reviewers whose identity is concealed may be more comfortable and so more open in their opinions about a piece of work than if they were asked for comments on a paper by a close colleague. Established authors get a fresh look at their work; newer authors can get their work before top experts in a field they might not otherwise be able to approach.
For the field, masked review offers the best chance to assess work based on its merits independent of the reputation and political influence of authors. This serves to promote the advancement of knowledge in a field based on the quality of the scholarship.
There are also disadvantages connected to the masked review process. Reviewers secure in their anonymity may prepare particularly stinging comments on papers, comments they would never offer to someone known to them. Reviewers may be tempted to offer only summative comments instead of suggestions for improving a paper. Reviewers may also lack certain information about the background of an investigation or the team engaged in the investigation, information that would allow them to make a better decision regarding revisions and publication. Reviewers of manuscripts moving through a masked review process often have only a decontextualized view of the line of research in question, or perhaps worse, reviewers who incorrectly guess the identity of the author or authors of a paper have inaccurate information about such matters. Moreover, reviewers may be just as inappropriately biased by the content of the manuscript as by knowledge of the identity of the author, for example, tending to reject papers with findings that challenge received views in a field.
Authors may use the cover of the masked review process to submit for review materials they feel uncomfortable sharing directly with colleagues. These unshared papers, when submitted, almost always end in rejection but consume editing and reviewing resources. Another problem arises when authors take the comments of an unrevealed reviewer more or less seriously than is warranted by the reviewer’s level of expertise. Of course, since the editor knows the identity of both the author and the reviewers, the editor can intervene to mitigate these kinds of problems.
These sorts of disadvantages of the masked review process have led to consideration of alternative strategies for having scholarly papers reviewed. One approach, suggested by Armstrong (1996), would entail a masked review process in which, in addition to the identity of the author, the results of a study were also concealed from reviewers to prevent them from rejecting studies that diverge from current thinking in a field. Armstrong suggests coupling this procedure with directions to reviewers that only permit them to recommend ways to improve a paper and discourage them from offering recommendations regarding publication, and with requirements that authors provide evidence that papers being submitted had been reviewed by colleagues prior to submission. These strategies attempt to shift the balance of effort in the review process toward the improvement of papers.
Another approach designed to foster improvement entails a process of developmental editing and reviewing. In this approach identities of authors and reviewers would be revealed throughout the process and an attempt would be made to cultivate a mentoring relationship between authors and their reviewers. In this way authors, editors, and reviewers could work together to improve papers submitted to journals.
The masked review process offers significant advantages to authors, reviewers, and the fields served by academic journals. Attempts to improve the process can be crafted to retain the current benefits while addressing some of the important limitations. Substantial improvements in the process will emerge just by having authors and reviewers become more aware of the goals and practices of the current masked review process.
- American Psychological Association. (1994). Publication manual (4th ed,). Washington, DC: Author.
- Armstrong, J. S. (1996, October). We need to rethink the editorial role of peer reviewers. The Chronicle of Higher Education, pp. B3-B4.