Analyzing Peer Review Practices of Systematic Review Search Methodology through Open Peer Review

Abstract

Introduction: Numerous studies have demonstrated the poor quality of systematic reviews. While many groups provide methodology recommendations to increase the quality of search strategies and compliance with reporting standards, it is unclear how often or how rigorously the search strategies are actually reviewed during the manuscript peer review process. The peer review process should address egregious methodology and reporting issues, but we hypothesize that this process seldom adequately addresses the search methods. Through Open Peer Review (OPR), we were able to investigate how peer review handles the search process of systematic review studies.

Methods: We conducted a search through one publisher's 54 medicine and public health journals that provide OPR documentation in order to identify systematic review papers published in 2017. For each article we determined if OPR data, reviewer and author comments, were accessible. If so, we assessed the search methodology and reporting quality of the search process with a grading rubric based on PRISMA and PRESS standards, and then mined peer reviewer comments for references to the search methodology.