Accept, reject, or revise? Improving scholarship by improving peer review

By: Elizabeth S. Karlin, MA, Staff Editor, Academic Medicine; Jennifer Campi, Senior Staff Editor, Academic Medicine; and Mary Beth DeVilbiss, Managing Editor, Academic Medicine

Peer review is the assessment (or review) of research by a scholar (a peer) who has expertise in the research topic or possibly the methodology applied. The purpose of peer reviews—the focus of this article—is to help ensure that the published literature is of the highest possible quality. Reviewers act as a sort of jury, determining if the research is of a quality that deserves dissemination.

For centuries, the peer review process has remained relatively unchanged: Researchers prepare a manuscript, a report of the study or investigation they have conducted. Next, the researchers send the manuscript to an appropriate journal. Usually, the manuscript undergoes an internal review. A decision-making editor (an editor-in-chief or associate editor) decides whether to reject the manuscript before review or to send it out for peer review. Decision-making editors, journal board members, and even the manuscript authors themselves may suggest reviewers. On average, two or three reviewers are selected from a pool of experts who are often authors themselves.

This next step in the process, review by a peer, is an essential element of scholarship. Although editors need not (and do not) always agree with or abide by reviewers’ recommendations, they rely on reviewers as a vital source of information about manuscripts. Editors and readers, other scholars, and the people who benefit from advances in science—that is, all of us—rely on peer reviewers to provide insights into research, to outline strengths and weaknesses, to uncover critical flaws, and to illuminate new discoveries.

The most important questions peer reviewers can answer are:

• Will this manuscript advance the literature? (Is the topic important? Will it change how we think about an issue? Does the research contribute to the field?)
• Is there a fatal flaw in the design (the methodology) or the argument (the logic)?
• How can the author/s improve this manuscript?

To answer the questions above, peer reviewers should examine the report’s introduction and reference list, the methodology, the results or findings, and the discussion and conclusion. We have provided a few questions and suggestions about each of these immediately below, followed by a description of other resources to help reviewers.

Introduction and references

Do the authors provide enough background information, based in the literature, for readers to understand the nature of the topic, the context, and the need for the current report? Is the literature the authors cite comprehensive, including both the most current articles as well as foundational research? Is the literature balanced, presenting different perspectives? Do the authors state their purpose, question, and/or hypotheses clearly?

Methodology

Is the method appropriate for the problem or question the authors hope to study? Have the authors provided enough details to allow future researchers to replicate their study? Is the setting clear? Where and when did the research take place? Who are the research subjects or participants? Did the authors procure all necessary ethical approvals, especially if working with human or animal subjects? Have the authors described their tools and their analyses? Have the authors considered all aspects of the problem?

Results

Do the results reflect the methodology? Do the authors report the findings of all their tools and analyses? Have the authors shown their results to be significant—statistically and clinically?

Discussion and conclusions

Have the authors considered whether their findings are generalizable to other settings, other subjects? Have the authors considered all of the implications of their findings? Have they shown how their findings will affect science? Have they considered avenues for, or questions to address in, future research? Have they discussed any limitations and the effects of those limitations on their conclusions?

Although the questions in the preceding paragraphs will help guide reviewers, here are some additional resources:

• Review Criteria for Research Manuscripts, 2nd Edition is a comprehensive guide for reviewers that outlines the entire review process and systematically examines each aspect of research that reviewers should assess, from title and byline, through abstract and intro, to conclusions and references. The resource was edited by two of Academic Medicine’s associate editors and written by experienced scholars and reviewers. It includes a useful checklist for reviewers; discusses new types of manuscripts such as reports of qualitative research, systematic reviews, and descriptions of innovations; and it even covers reviewer etiquette. The manual can serve as a useful tool for novice and experienced reviewers alike. It will help any reviewer struggling with a difficult manuscript, will assist writers who want to prepare the strongest possible manuscript, and will support trainees (and their mentors) who are learning how to conduct, assess, apply, disseminate, and discuss research.
• What editors want: An overview for reviewers answers a common reviewer question: What do editors want? This short video presentation by the editors and staff of Academic Medicine covers reviewer etiquette, review format, and review content. It is part of the journal’s expanding collection of reviewer resources, which include a downloadable guide to reviewer recommendations and are available through the Academic Medicine’s For Reviewers page. Additional videos and practice review exercises are in development.
• Reviewer Workshops – During these dynamic, interactive workshops, Academic Medicine’s editors and staff present basic information on peer review and then turn the session over to the participants. Session attendees work in small groups to review an actual manuscript submitted to Academic Medicine. At the end, the presenters and attendees discuss the manuscript, the process of reviewing it, and the insights gained through the process. Contact the journal at [email protected] to learn about upcoming workshops and events.
• Reviewer Resources Series on Academic Medicine’s Blog, AM Rounds – Perhaps one of the best ways to learn about and improve peer review is to read what the peer reviewers themselves have to say. In this series of 11 blog posts, reviewers who have won Academic Medicine’s Excellence in Reviewing Award share their tips on how to conduct and write quality peer reviews, their thoughts on the benefits and purpose of peer review, and even their love of the process.

And here—enjoying peer review, the benefits of peer review for the reviewers and authors—is where we want to end. In addition to the pragmatic benefits of review, including letters recognizing service, lines on a CV, and sometimes rewards or CME credit, scholars note other reasons for continuing to review one another’s research. Despite the time and effort that quality reviews require, scholars assess manuscripts because doing so allows them the opportunity to learn about the newest discoveries in their field. They are participating in the scholarly process, giving back to the community that has helped them, and advancing science for the benefit of us all.