For the Record: Preparing Reviews
by Gary Natriello - 2000
The elements of reviews that will be useful to journal editors
The quality of an academic journal is heavily dependent upon the work of the scholars asked to review the materials submitted for publication. Collectively, reviewers do an enormous amount of work in reading papers, offering suggestions to authors, and giving advice on publication decisions to editors. Reviewing is a task that many of us will be asked to do, but few of us are given any training or guidance on how to prepare a review that will be useful to an editor and an author. In this primer on preparing reviews of papers submitted to journals, I want to discuss some key components of reviews that are useful to authors and editors.
After editing two major journals for over eight years now, I have read thousands of reviews. The most useful reviews make the tasks of editors and authors markedly easier. In developing the advice that follows I am compiling the best features of the reviews I have read and used; it is rare for a single review to contain all of the elements I will describe. Nevertheless, if more reviews contained these elements, revising and editing papers would be easier, and our journals would be stronger.
Most academic journal editors ask reviewers for three kinds of comments. First, they ask reviewers to prepare comments on the paper that can be sent directly to the author. Authors expect to receive such comments after waiting months to hear from a journal. Second, reviewers are also invited to prepare comments that will only be seen by an editor. Third, reviewers are asked to make a recommendation regarding publication. Each of these types of comments plays an important role in the overall review process, and I will consider each in turn.
The comments prepared for authors typically require the most time and attention on the part of reviewers. Because these comments usually contain the most detailed discussion of the paper, it is often best to prepare them first. Reviewer comments on a particular paper might begin with a paragraph that provides a brief summary of the paper as seen by the reviewer. In addition to a few sentences describing the paper, there might be some discussion of how the paper fits into a field or an area of study and how the work relates to or expands upon other work in the field. In the same paragraph the reviewer might offer an opinion as to whether the paper breaks new ground, provides needed confirmation of earlier findings, or if it adds little to what is already established. This introductory discussion can also highlight any other strengths of the paper. This paragraph serves to alert the author to the reviewer’s perspective on the paper. When the same paragraph is read by an editor, it provides an orientation to a particular paper among the many that an editor may be dealing with simultaneously.
Following the initial overview of the paper, the reviewer might offer critical comments on the major elements of the paper. These elements will differ for different kinds of papers, but for present purposes I will consider the five elements characteristic of traditional reports of empirical research: introduction and statement of the problem, review of the relevant literature, methods, results, and implications and conclusions. Reviewers can assess the strengths and weaknesses of the paper in these five areas. For each area I will note only the major issues that might guide the reviewer.
The introduction of the paper should provide a clear statement of the topic of the paper and the problem that it will examine. Reviewers often note when a paper lacks focus or when the purpose of the paper stated in the introduction is not carried through to the end. Reviewers also sometimes comment on the importance of the intended focus of the paper.
Most scholarly papers attempt to make connections to other scholarly work that is relevant to the topic at hand. Sometimes this involves a full-scale review of the literature; at other times it involves identification of just the key pieces of scholarship guiding the current effort. Reviewers often note when the literature cited is dated or not fully representative of work in the field. Reviewers can provide great assistance to authors and editors by noting when key pieces of the literature in an area have been ignored. Providing the publication details on other relevant studies is very helpful to authors s they develop a paper further.
For papers reporting on empirical investigations the discussion of methods is very important; a complete description of the methods of a study is necessary to allow others to evaluate and perhaps replicate the study. Reviewers are quick to identify missing information about the sample or the study procedures. If complete details on all elements of the study cannot be included, then reviewers often expect references to technical reports containing the additional information. When study procedures are described fully, reviewers can judge their technical adequacy and assure themselves that the methods selected were the most appropriate for the study at hand.
Presenting the results of a study can involve various strategies and include a narrative along with tables and graphs. Reviewers are concerned that the results be clearly and completely reported so that readers gain a full understanding of the findings. Reviewers sometimes provide authors with important information to guide revisions to make the presentation of findings clearer to readers.
Discussing the implications of an investigation and developing conclusions require careful attention to the original problem posed for the study and the results obtained. Reviewers are quick to point out when the implications and conclusions drawn by an author move far beyond the findings presented. Reviewers are also often able to tell authors how their work may be connected to larger issues of research, policy, and practice.
In addition to these five areas of a report of an empirical study, reviewers may find themselves offering commentary on a wide range of other papers submitted to journals. Essays on broad topics in a field or reviews and compilations of the literature require reviewers to focus their efforts in different ways, but all reviews should include commentary on the major sections of a paper, or at least those sections deemed in need of additional work.
In addition to the initial descriptive paragraph and the comments on the major sections of the paper, reviewers can provide very specific comments on the writing including typos and other minor errors when they notice them. Such comments are always welcome even though copyediting will typically catch most of these errors.There are several things that should not be included in the comments prepared for the authors of papers. Particularly harsh comments or comments that impugn the author should be avoided since they serve no useful purpose. Editors sometimes remove these disparaging remarks before passing the comments onto authors, but it would be easier if reviewers refrained from inserting them in the first place. Reviewers should also refrain from making comments that reveal their identity if the review process is intended to be anonymous. Reviewers who find some reason to convey their identity can include a note to the editor offering to reveal themselves and let the editor decide if it is appropriate. Reviewers should also not include a recommendation regarding publication in their comments to authors. Such recommendations, whether positive or negative, can be misleading to authors when they receive them without the benefit of an editor’s review.
The comments prepared for the editor might include all of the materials prepared for the author. Sometimes reviewers simply copy the comments prepared for the author onto the form provided for comments for the editor; at other times they include a note referring to the author comment form. In addition, the reviewer might include other comments directed only to the editor. Such comments could include things that might be too harsh to convey to the author directly. For example, if a reviewer believes that the level of sophistication of the author is not sufficient to succeed in the necessary revision of the paper, the reviewer might write “I doubt that the author of this paper will be able to accomplish the necessary revisions.” Reviewers might also want to indicate to editors that a particular paper is outside their immediate area of expertise. Reviewers might also want to express the strength of any recommendations regarding publication that they wish to make. Reviewers might note the importance of the article and its appropriateness for the particular journal and its audience.
The final element in a report from a reviewer is a page with a limited set of options regarding the publishing decision, typically ranging from “Publish as is” to “Do not Publish”. It is important for reviewers to indicate clearly which of the various options they recommend. It is all too easy for a recommendation imbedded in a narrative report to the editor to be lost or misinterpreted. Completing the fixed set of options places the reviewer’s recommendation clearly on the record. This is particularly important for editors who are often trying to reconcile different reactions from reviewers.
Developing a final recommendation is a major responsibility for a reviewer. In formulating such a recommendation reviewers typically have little difficulty deciding that something should be published in its current form; most reviewers make these rare decisions with apparent ease. They have substantially greater difficulty deciding whether an author should be invited to revise and resubmit a paper or whether the paper should be rejected. There are several questions that reviewers might consider in making such a decision. First, a paper should only be recommended for revision if the objections raised in the review can be addressed given the available data. Second, a paper should only be recommended for revision if the reviewer believes that the author is capable of carrying out the revisions to produce a publishable paper. It is not helpful to either the author or the editor to encourage a revision if the likelihood of eventual publication in the journal is remote. The author’s interests in particular would be better served by a recommendation to reject the paper so that it might be submitted to another journal.
The work of reviewers is involves substantial effort. Strengthening that work will make it more useful to editors, authors, and the field at large.