Bibliography and Summary:
Electronic Peer Review Management

A report prepared by Kam Shapiro for the Scholarly Publishing Office


Outline:


Introduction
1. General list of functions for online peer review tools
2. General User Issues
3. Summaries of articles on automated peer review processes
4. List of Software Tools
5. List of Journals using automated peer review
6. Other resources
7. Editor interviews

Introduction: Electronic Peer Review:


A variety of software tools are now available that enable the electronic management of peer review processes for scholarly journals. These tools promise to facilitate efficient and centralized control and/or supervision by journal staff of the submission, assignment, tracking and publication of articles though the web, as well as enabling a central archive of various tasks performed. Some programs keep all texts in on-line format throughout these processes, using multiple windows to allow reading, editing and on-line publication of articles, while others use automated ftp and email processes to exchange documents in standard formats. The specialized features available vary widely, but the more highly developed programs share many characteristics. Generally speaking, the programs consist of several interfaces: An author screen allows individuals to electronically submit articles (format pre-specified or automatically converted); an editor screen allows editors to identify, read and notify or assign submitted articles to potential reviewers; a reviewer screen enables reviewers to accept and read or receive articles, post or send their comments and suggested revisions to either editor or author. The ability to customize these processes and interfaces varies significantly among different tools. For example, some allow for the selection of blind or double-blind review and editing processes, some allow an editor to assign articles to reviewers by selecting from an on-line list, or in some cases even have articles assigned automatically by category of specialization (see Bepress, below).
Despite their different “bells and whistles,” the workflow process is relatively constant across most of the software reviewed here. This no doubt reflects the relatively standard elements of peer review processes across multiple journals. That said, a primary concern for editors is the adaptability of any software to the idiosyncrasies of their process. However, editors may not appreciate the extent to which their processes resemble those of other journals.


General list of functions for online peer review tools


1. Automated Submission – provides templates/instructions for submission, converts and uploads content from authors into html/pdf/other format.

2. Automated Notifications – Generates email to editors and reviewers and authors notifying them of articles to be reviewed, reviews or edited copies available online, etc.

3. Article assignment/tracking, Event Logging - retrieves list of appropriate/available editors and or reviewers and tracks those who choose or are assigned to particular articles – making it easy to check on status of reviews (completed or pending). Keeps log of actions performed (such as revise/resubmit requests). Some programs can automate the assignment of reviewers based on article categories. [Few journals are likely to adopt the last. Typically, an editor in chief (EIC) or associate editors will nominate reviewers. Once this is done, however, the software can contact reviewers from this list and alert the EIC to their acceptance of articles.]

4. Reviewing/Copyediting – Provides screens for writing and saving or sending finished review to editor. Usually involves split screen allowing reviewer to view and excerpt from article while writing review. Also may allow editing of article content by assigned authors, editors or reviewers (sometimes with flexible authorization) and makes revised version available to other editors and or authors (with or without anonymity). Some tools preserve an original copy of the submitted article in a separate file automatically. [Note: None of the tools examined here included the ability to track changes in texts]
4.1 Flexible Authorization – Allows journal staff/EIC to make articles or reviews in process available to different users.

5. Quality/Category tags – Provides standardized tags with which editors can mark pre-print articles for quality and proper classification.

6. Blind/Doubleblind Option – Enables the anonymity of authors and/or correspondence between reviewers and authors (removing name and acknowledgements, etc.). If both can view articles and reviews in online work space this can save editors significant time and effort. Editors may, however, wish to excise confidential portions of reviews prior to allowing authors access to them.

7. Time reminders or enforcement (“nagging” – see Conservation Ecology or Fee assignment - Bepress) – establishes deadlines for submission of reviews and automatically sends email reminders to editors and reviewers. Alerts editors of completed, pending or overdue reviews.

8. Posting – Automatically formats and publishes articles that have received a proper number and quality of reviews. May also notify subscribers (see consecol).

9. Reviewer Information and Performance Tracking – Maintains accessible information on past performance, contact info., availability, etc. and alerts editors when they are selected for a new review.

General User Issues


Flexibility - Customization of automated peer review processes for different journals.

Examples:



Suggestion (from conversation with Bepress): Journal staff often presume their processes to be more idiosyncratic than they appear once compared with those of other journals. Review procedures are relatively constant across many journals.

Confidentiality – With online content, confidentiality issues become a concern. Editor control over who may access article and reviews at different stages can be a somewhat complicated matter. For instance, an Editor may wish to make the text of a review or a revise/resubmit recommendation available to an author, but may first wish to excise some comments.

Platform Independence – Note: This is a primary concern for journals as well as a point of contention in discussions of the viability of automated peer review.

Editing and Revision

Security: Some tools use encrypting technology (SSL) for document transfer and correspondence.


Summary of articles on scholarly electronic peer review processes:

Discussions of automated peer review tools for scholarly electronic publications generally focus on several key issues and advantages:

More general discussions of online peer-review processes focus on providing a means of filtering and classifying the otherwise unmanageable scope and quality variation of online content.

Articles:
http://www.consecol.org/Journal/consortium.html Conservation Ecology Project: Discussion of their own online submission and review software, currently in use. Provides a detailed discussion of the review tools and process (includes double-blind submission and review). Site also includes a demonstration of the process. See the discussion of upgrade plans and associated issues and requirements for the software.
Note: They are seeking collaboration with other entities - Contact Info: “Please contact Shealagh Pope, the Project Coordinator for the Conservation Ecology Project, at sepope@consecol.org if you have any comments or questions”
See also a discussion of software function upgrade plans and intentions for distribution at http://www.resalliance.org/consortium/consortium.html Includes plans for SGML/XML coding of accepted articles.

http://www.press.umich.edu/jep/05-04/edmonds.html
A Proposal for the Establishment of Review Boards BRUCE EDMONDS
Proposal includes graphs of different existing review processes and a more complex system for automatically assigning material in specific topic categories to appropriate reviewers.

http://www.firstmonday.dk/issues/issue4_4/proberts/ General discussion of the value of online publication and role of peer review of scholarly work focusing on time, storage space, and cost savings.

http://www.press.umich.edu/jep/05-04/sheridan.html ‘Digital Workflow: Managing the Process Electronically’ – Article outlines the features and extols the virtues of online scholarly publication, including automated peer review management (“An additional benefit might be that they give the appearance of reducing bias because they are systematized”).

http://www.press.umich.edu/jep/03-03/vanreenen.html
Manifesto of sorts advocating library-sponsored alternatives to commercial publishing. Depicts a battle over electronic subscription licensing (esp. of scientific and technology resources) with “predatory” commercial enterprises. Mentions electronic peer review as an important offering to potential users. Language conforms with the “learning by doing” motto of Conservation Ecology site.

http://www5conf.inria.fr/fich_html/papers/P55/Overview.html
"Electronic Management of the Peer Review Process", G. Jason Mathews and Barry E. Jacobs.
Article from 5th International www conference.
“This paper addresses the electronic peer review process problem. Namely, how does one electronically manage the complex process of peer reviewing papers over a physically distributed set of participants (i.e., authors, reviewers, and administrators).” - Discusses technical dimensions of “Electronic Management System”. Describes programming tasks for various peer review functions. Provides examples of interactive forms and requisite code.

http://www.bodley.ox.ac.uk/icsu/donovanppr.htm
"The Truth About Peer Review", Bernard Donovan
Skeptical discussion of the value of electronic peer review of scientific journals, focusing on cost savings, effects on academic community, etc. “In theory, electronic refereeing offers many benefits, and would be readily accepted by the academic community, for 63% of the authors canvassed in a study done by the ESPERE project (Wood,1997)” Article emphasizes the difficulties of assimilating multiple text and graphics formats, as well as the uneven technical capacity of users to view and print complex documents.

http://www.ariadne.ac.uk/issue5/jime/ Article describes an online “argumentation” and review process for pre-print publications and advocates interactive forms of online publication. Cites conservation ecology as primary example of peer review software.

Peer Review Software tools


http://eos.wdcb.rssi.ru/tools/pros.html Free (for noncommercial use) online review tool (“pros”). Demo available from this page. Somewhat limited functionality.

http://www.bepress.com/
PR tool with flexible and extensive submission, assignment and tracking. Good interface design. Includes an optional “banking” system whereby authors accrue“debts” of 3 reviews per submission with a buyout option (demo set at $1000).

http://peerreview.temple.edu/ “Temple Peer Review Manager” – Developed at Temple University. Flexible management system for assigning different editors. Special feature allows blind or open submission and an “open restricted” correspondence system that allows authors and reviewers to communicate anonymously. Contact info: “Please send email questions and comments to Dr. Munir Mandviwalla at mandviwa@temple.edu or 215-204-8172.”

http://www.consecol.org/Journal/consortium.html Conservation Ecology Project
See discussion under Articles, above. Software still being developed and enhanced. Future editions plan to convert documents to XML, SGML, etc.

http://www.scholarone.com/applications.html ScholarOne
Commercial submission, editing and peer review software. Description and demo available. Extensive list of functions. Submission upload feature promises ‘on-the-fly’ conversion of texts and images (the latter to jpeg format). Contact info:
Send e-mail to getstarted@ScholarOne.com or call (804) 817-2040 ext. 118.

http://cjs.cadmus.com/rapidreview/index.html ‘Rapid Review’ – Commercial submission and review tool. Extensive functions. See their list of 24 on their site. Includes elaborate tracking and report functions, on-line help system, automatic recognition of user status with different levels of security, etc. For a personal demonstration of Rapid Review, please contact Lisande Bissonette. mailto:bissonettel@cadmus.com

http://www.biomedcentral.com/info/niche.asp Biomed Central: Central host for biomed journals offering electronic peer review tools for journals wishing to publish through them. Central retains ownership of journals published.

http://www.ukoln.ac.uk/services/elib/projects/espere/ ESPERE project: “The experiment is intended to achieve an introductory level of article submission and peer review by email transfer of a file which includes all the figures and tables applicable to the paper.”

http://catalyst.washington.edu/home.html Catalyst Peer Review Tool: A web-based system for student collaboration and comments on others’ work. (not a publishing tool)


Comparative list of features

  Bepress PROS Temple ConsEco Scholar1 RapidRev
             
1. Automated Submission X X X X X X
2. Automatic Notifications X   X X X X
3. Artilce assignment/tracking X X X X X X
3.1 Event Logging ? ? X X X X
4. Reviewing/Copyediting X X X X X X
5. Quality/Category Tags X         X
6. Blind/Doubleblind Option Blind Auth   X X ? X
7. Time reminders or enforcement Flexible   Flexible Flexible X X
8. Automatic Posting X   X X X X
9. Reviewer Info/Performance ?   ? X X X
10. Security ? ? SSL ? X Flexible

Online Journals with electronic submission/review:


http://www.the-aps.org/staff_directory.htm American Physiological Society - Includes list of multiple science journals with central submission that all use the ScholarOne system.

http://www.endocrinology.org/sfe/onlsub.htm Uses the ‘ESPERE’ system for online submission.

http://www.rsc.org/is/journals/current/PhysChemComm/pccpub.htm Uses electronic referee system (‘Referee Report Form’)

http://www.neuroguide.com/submission.html “Neurosciences on the Internet” Online submission (no mention of electronic peer review system).



Other Resources


http://www.public.iastate.edu/~CYBERSTACKS/EJI.htm#Accelerated “A Registry of Innovative E-Journal Features, Functionalities, and Content” - List of sources on electronic publishing, including peer review processes.

http://www.espere.org/links.htm “This page has links to a number of interesting sites who are active in various aspects of online submission and/or peer review.”

http://info.lib.uh.edu/sepb/sepb.html Scholarly Electronic Publishing Bibliography
This selective bibliography presents over
1,350 articles, books, electronic documents, and other sources
that are useful in understanding scholarly electronic publishing
efforts on the Internet and other networks.


http://www.sun.com/products-n-solutions/edu/libraries/digitaltoolkit.html
Sun Microsystems, Inc. has published the second edition of its
popular "Digital Library Toolkit", a valuable resource for anyone
planning a digital collection. To download a free copy, go to:

Interview notes

John King


John explained, first of all, that journals should be thought of primarily as a forum for discussion among academic peers. More specifically, he provided an extensive analysis of different types of journals and corresponding priorities for information “flow.”

Types of Journals:
“Frontwave” – Journals devoted to new science and evolving discourse.
These journals require limited peer review since authors and their work are likely already known within a specialized community. The validity of their experiments and results for the purposes of publication, he suggests, are relatively easy to determine in this context.

“Reformulation, Re-thinking” – Primarily social science and humanities journals that serve the purpose of a public forum for debate. As their terms are more contested, these journals generally require more reviews (often three for tie-breaking purposes).

“Position/Action Journals” – Policy-oriented journals. [We did not discuss their procedural needs]

Other issues: Quality control, disciplinary boundaries (editors may need to choose reviewers carefully so as not to elicit a hostile response based on sub-disciplinary antagonisms that cross a given topic of study).

Reviews may vary in tone: Those of work by junior faculty can less afford to be hostile, as it can discouraged them.

John also provided a general outline of review process:

“Case Management:” Basically, the tracking of cases (articles) through a process with a discrete start and end (submission and publication). Most issues come down to keeping track of all cases in process and avoiding lost articles. This involves reminders and applying pressure to associate editors, authors (esp. when asked to revise and resubmit) and reviewers (to release article in a timely manner).

“Workload Management:”

This involve decisions regarding the role of different members of the journal staff. For example, associate editors may be charged with different articles and reviews in their respective fields of expertise.
Authors may pick editors that are most suited to their area and submit directly to them.
Or, there may be a centralize intake, after which articles are distributed (John strongly prefers centralized intake so as not to lose track of articles – the “black hole” problem, as he put it).

Issues and Problems:
Level of involvement by Editor in Chief: Does the article come back to the center at every stage in the process (after initial review, prior to revise/resubmit, etc.)?

Rejection of article by editor regardless of reviews. Sometimes an editor may wish to be able to over-ride positive (or negative) reviews.

Electronic correspondence: Issues of version control and formatting.

Policy regarding access of authors to manuscripts in progress (some may continue to edit their work indefinitely if given the chance, he notes).

John discussed the difficulties involved in setting up a system that allows associate editors to manage different types of articles separately and then get them all together prior to publication- suggested the possibility of using a client server database [he was not sure if this was technically feasible].

David Velleman


David expressed interest in automated review tools, but had several concerns about compatibility and ease of use.

Compatibility- Philosopher’s imprint asks authors to submit rtf versions of articles and then allows referees to open them in whatever format they prefer. Nonetheless, he stressed that it is important that authors be able to see the formatting and style in which their article was to be viewed by referees. At present, this happens in the final stage of the publication process, when final proofs of articles (as they would appear in print) are sent to authors as hard copies for them to mark prior to actual publication. This ability for authors to see the final appearance of their work and for editors to see their marks on such a copy is very important for the journal.

Ease of use – David emphasized that the interface for reviewers in particular must be “obvious,” since their participation is already a courtesy.

PI uses two reviewers, typically, with a third brought in for tie-breaking purposes.

David welcomed the idea of a double-blind online conversation between authors and referees in which each was able to see both article and review in an online work space. He would like to be able to do this, but would also like to be able to censor privileged correspondence between reviewer and editor (himself) prior to making the review available to the author.