Peer Review: Who Is a Peer and How Should One Behave?
by Ross E. Mitchell - May 27, 2002
Suggestions for broader participation and greater efficiency in the peer review process while preserving quality and legitimacy.
Not too long ago, the Executive Editor of Teachers College Record (TCR) shared his thoughts and concerns about peer review in a "For the Record" column (Natriello, 1999). In this commentary, I wish to discuss two points related to peer review. First, I propose that we examine more fully who counts as a peer. As a community of scholars, we need to be aware of our diversity and put our many talents and perspectives to greater use by soliciting fuller participation. Second, I believe we need to be explicit and self-critical concerning how peer review is practiced. Claims of status, privilege, and obligation remain largely implicit and unexamined. My hope is that by continuing to scrutinize the practice of peer review the community of scholars will attend to both the productive matters of quality and efficiency and the institutional concerns of access and legitimacy.
Before developing my own arguments, let me be sure we are all aware of the essay to which I am responding. In the original "Time and the Review Process," the readers of TCR were offered a delightfully honest portrayal of peer review. The essay addressed the problems introduced by the delay between submission and publication, reasons for the delays, potential solutions to the delay problem, and some justification for delays. In order to keep this commentary brief, I will not review all the details here. However, I would like to draw out one point worthy of emphasis. I was particularly pleased to see that the editor demonstrated a genuine awareness that the current norm for publication review is out of step with the rising expectations imposed on the most recent generation of social science research scholars, particularly tenure-track faculty members. I fully concur with his statement that, "less senior scholars are often under considerable pressure to have their work published prior to personnel reviews with implications for reappointment and tenure decisions" (pp. 697-698).
WHO COUNTS AS A PEER?
I now offer some thoughts about the reviewing norms in the column that seemed to me less worthy of attention. I am not saying that anything included was not wholly worthy, just that I differ with the author's priorities. I begin with the matter of who participates in peer review, particularly since participation influences the length of the delay in the process. That is, I am proceeding on the assumption that increasing the number of available reviewers would improve the efficiency of the process by shortening the queue and improving the probability that reviewers would respond more quickly. However, I must acknowledge that efforts to expand participation increase the likelihood that the overall quality of reviews will be diluted. In the two points I offer to the discussion of who counts as a peer, potential threats to quality in search of increased efficiency are addressed.
First, seeking only recognized "experts" in order to maintain quality is not a flawless strategy. I suggest that there is a delicate balance between expert ratification and intellectual gatekeeping. It is important to decide whether or not the ideal of a dispassionate and skeptical stance is truly achieved in the review process or if real findings are at times dismissed by biased judgments based upon a limited or elitist intellectual heritage. (As the class size research debate has evidenced, for example, one man's expert can be another man's hack.) A possible, though risky, solution is to include more junior scholars in the review process. My hesitation here has to do with the fact that a junior scholar may lack the necessary confidence to respond in a timely manner because s/he feels a need to do some homework before writing the review (not a bad sentiment, and possibly one shared by senior scholars as well). Nonetheless, if these new folks are worthy of tenure-track appointments they should also be competent to evaluate the ongoing production of scholarly manuscripts (and do not forget the full-time researchers at places like RAND, AIR, and the various federal agencies).
Second, and related to the first point, distributing the reviewing load is certainly going to be easier if the potential reviewer list is longer. Journal editors might also want to consider looking for the serious consumers of scholarship at teaching colleges and universities. These folks are likely to read widely and invest a lot of energy in interpreting research for those who are interested in results but are not experts themselves. Again, there are risks associated with reviewers who are not skilled in the various technical crafts of research production, but quality control is a consumer issue as well as a production issue. By drawing from both research and teaching institutions, and both junior and senior scholars, participation can be increased, fairly expanding the definition of who counts as a peer. Threats to the quality of the peer-reviewed literature are not severe, and this strategy responds to the pressing need for improved efficiency.
HOW IS PEER REVIEW PRACTICED?
I now turn my attention to three points about the current construction of the peer-review process as represented in the TCR editorial essay. The rationale offered for declaring that there are "benefits of a lengthy review process" (p. 701) is problematic. My concern depends on a fundamentally conditional statement: "A long careful review process can lead to the improvement of the quality of academic publications if reviewers and editors take the time to examine papers carefully and if authors take the advice they are given to revise their work prior to publication" (p. 701). The matter of care and the taking of time and advice require examination.
The word "careful" is important. Just like in medicine, for certain cases, the nature of the care received can do as much or more harm than the condition that requires care. Without clear guidelines for the standards of care and some "policing" of the practice, careful is a loaded qualifier. My point here is not just worrying about how things are done; this is a matter subject to empirical investigation. We need to discern accurately what standard of care is currently practiced and whether or not it is adequately conceived or even appropriate.
The phrase "take the time" is highly problematic. There is a big difference between taking time in terms of finally getting around to reviewing a manuscript sitting about for four months, and taking time in terms of reflecting and responding. Though this retraces the path followed earlier in the original editorial, this is a point requiring empirical verification. The standard assumption seems to be, based upon the length and sophistication of comments, as well as the protests of being busy and overworked, that when reviews come back they could not have been returned sooner. I suggest that this requires examination. Further, through examination, we may very well discover that this whole process has been allowed to drag out because of norms of self-interest (i.e., the marginal return on providing a timely review is too low) rather than real limitations on time. Unfortunately, if such an hypothesis were true, it is wholly indeterminate whether such a realization would lead to shortening the time for review or reducing participation in the process. But just like American foreign policy in the Middle East, deference to immoral regimes seems to be an indefensible position. (This last statement is overly strong, but I am searching for an analogy that helps to reinvigorate the notion of a community of scholars.)
The phrase "take the advice" in the same sentence with "prior to publication" is not helpful. Am I supposed to believe that an author who does not favorably respond to reviewer and editor advice is ever going to get a manuscript published? The point about taking advice is very important if an author is worried about how long it takes to advance from revision to publication, but this is fundamentally a power question. There needs to be some honesty about the fact that authors are beholden to an editor's judgment. An editor needs to clearly communicate to authors the firmness of his/her decision about the request for revisions. This way, authors know whether they may be investing or wasting time in making the case that their presentation in the original manuscript is better than the suggested revisions.
The critique of peer review as currently practiced is simple. A lengthy review process does not hold any benefits by itself. There are critical contingencies. The editors, reviewers, and authors have to responsibly negotiate a settlement as to what constitutes acceptable contributions to the archival literature. Explicitly acknowledging power relations and making clear the difference between expressed opinions and genuine demands would go a long way toward ensuring that all participants understand what constitutes standard practice and what is idiosyncratic to the manuscript under consideration.
GUARDIAN OR ADVOCATE
In closing, I would like to thank the executive editor of TCR for claiming authorship and exposing himself to criticism. In "Time and the Review Process," the boundary between advocacy for change and guardianship of the existing order was carefully crossed more than once. It seems that editors occupy the unenviable position of having to maintain and justify a solidly institutionalized practice, often whether they like it or not. Certainly, the peer-review process is an important tradition, but it requires invigoration. Both participation and practice standards need to be examined. Despite fractious debates and paradigmatic conflicts, peer review depends on being able to locate a community of scholars. I propose that we attend not only to improving efficiency while maintaining quality, but that we concern ourselves with access to and the legitimacy of the peer review process as well. Hopefully, a continuing conversation about the review process will not only help to meet the individual needs of scholars seeking to publish their work, but sustain a dynamic and vibrant exchange among authors and the full range of scholars who count as peers.
Natriello, Gary. 1999. "For the Record: Time and the Review Process." Teachers College Record 100(4): 697-701.