Bibliography and Summary:
Electronic Peer Review Management
A report prepared by Kam Shapiro for the Scholarly Publishing
Office
Outline:
Introduction
1. General list of functions for online peer review tools
2. General User Issues
3. Summaries of articles on automated peer review processes
4. List of Software Tools
5. List of Journals using automated peer review
6. Other resources
7. Editor interviews
Introduction: Electronic Peer Review:
A variety of software tools are now available that enable the
electronic management of peer review processes for scholarly
journals. These tools promise to facilitate efficient and
centralized control and/or supervision by journal staff of the
submission, assignment, tracking and publication of articles though
the web, as well as enabling a central archive of various tasks
performed. Some programs keep all texts in on-line format
throughout these processes, using multiple windows to allow
reading, editing and on-line publication of articles, while others
use automated ftp and email processes to exchange documents in
standard formats. The specialized features available vary widely,
but the more highly developed programs share many characteristics.
Generally speaking, the programs consist of several interfaces: An
author screen allows individuals to electronically submit articles
(format pre-specified or automatically converted); an editor screen
allows editors to identify, read and notify or assign submitted
articles to potential reviewers; a reviewer screen enables
reviewers to accept and read or receive articles, post or send
their comments and suggested revisions to either editor or author.
The ability to customize these processes and interfaces varies
significantly among different tools. For example, some allow for
the selection of blind or double-blind review and editing
processes, some allow an editor to assign articles to reviewers by
selecting from an on-line list, or in some cases even have articles
assigned automatically by category of specialization (see Bepress,
below).
Despite their different “bells and whistles,” the
workflow process is relatively constant across most of the software
reviewed here. This no doubt reflects the relatively standard
elements of peer review processes across multiple journals. That
said, a primary concern for editors is the adaptability of any
software to the idiosyncrasies of their process. However, editors
may not appreciate the extent to which their processes resemble
those of other journals.
General list of functions for online peer
review tools
1. Automated Submission – provides
templates/instructions for submission, converts and uploads content
from authors into html/pdf/other format.
2. Automated Notifications – Generates email to
editors and reviewers and authors notifying them of articles to be
reviewed, reviews or edited copies available online, etc.
3. Article assignment/tracking, Event Logging - retrieves
list of appropriate/available editors and or reviewers and tracks
those who choose or are assigned to particular articles –
making it easy to check on status of reviews (completed or
pending). Keeps log of actions performed (such as revise/resubmit
requests). Some programs can automate the assignment of reviewers
based on article categories. [Few journals are likely to adopt the
last. Typically, an editor in chief (EIC) or associate editors will
nominate reviewers. Once this is done, however, the software can
contact reviewers from this list and alert the EIC to their
acceptance of articles.]
4. Reviewing/Copyediting – Provides screens for
writing and saving or sending finished review to editor. Usually
involves split screen allowing reviewer to view and excerpt from
article while writing review. Also may allow editing of article
content by assigned authors, editors or reviewers (sometimes with
flexible authorization) and makes revised version available to
other editors and or authors (with or without anonymity). Some
tools preserve an original copy of the submitted article in a
separate file automatically. [Note: None of the tools examined here
included the ability to track changes in texts]
4.1 Flexible Authorization – Allows journal staff/EIC
to make articles or reviews in process available to different
users.
5. Quality/Category tags – Provides standardized tags
with which editors can mark pre-print articles for quality and
proper classification.
6. Blind/Doubleblind Option – Enables the anonymity of
authors and/or correspondence between reviewers and authors
(removing name and acknowledgements, etc.). If both can view
articles and reviews in online work space this can save editors
significant time and effort. Editors may, however, wish to excise
confidential portions of reviews prior to allowing authors access
to them.
7. Time reminders or enforcement (“nagging” –
see Conservation Ecology or Fee assignment - Bepress) –
establishes deadlines for submission of reviews and automatically
sends email reminders to editors and reviewers. Alerts editors of
completed, pending or overdue reviews.
8. Posting – Automatically formats and publishes
articles that have received a proper number and quality of reviews.
May also notify subscribers (see consecol).
9. Reviewer Information and Performance Tracking –
Maintains accessible information on past performance, contact
info., availability, etc. and alerts editors when they are selected
for a new review.
General User
Issues
Flexibility - Customization of automated peer review
processes for different journals.
Examples:
- Submission and Correspondence Formatting – Different
journals may have widely varying requirements for submissions and
articles under review, some requiring a standard format or version,
others accepting a variety of text formats.
- Ability to authorize editing of content by authors, editors or
reviewers after article is ‘published.’
- Customized fields – Enable customized rating system for
reviewed articles. Allow reviewers to paste longer reviews into
comment fields.
- Ability to “push through” contents stalled in
process by absent reviewers, etc. – Rapid Review offers a
“proxy” function whereby journal staff can assume the
role of any user. [This poses little trouble for most systems,
since editors have open access to texts in progress].
- Number of Reviewers – Varies by type of journal.
Often three are used for “tiebreaking” purposes. With
systems that choose reviewers automatically (Bpress), flexibility
in this area becomes important.
- Editorial Authority – Some journals may allow editors to
reject articles despite positive reviews. In this case, they may
wish to flag the decision for inspection by EIC. – This need
not be incorporated in an automated process
- Level of involvement by EIC or journal staff – Article
may return to EIC after each step in process, such as revisions and
editing, or only after approval by associate editor(s).
- Associate Editors – Some journals may wish to use a
“distributed submission” process, whereby authors
submit articles directly to different editors. Presents tracking
challenges compared with central submission site.
- Ability to set different protocols for each associate editor
while tracking all articles centrally. This could be complex,
programming-wise. Client server model?
Suggestion (from conversation with Bepress): Journal staff often
presume their processes to be more idiosyncratic than they appear
once compared with those of other journals. Review procedures are
relatively constant across many journals.
Confidentiality – With online content, confidentiality
issues become a concern. Editor control over who may access article
and reviews at different stages can be a somewhat complicated
matter. For instance, an Editor may wish to make the text of a
review or a revise/resubmit recommendation available to an author,
but may first wish to excise some comments.
Platform Independence – Note: This is a primary
concern for journals as well as a point of contention in
discussions of the viability of automated peer review.
- Uploading Documents: Conversion of files (text and image) from
multiple formats and versions in process of submission/uploading.
Some journals use templates for submissions. Some require RTF
formatted attachments that reviewers can then open using whatever
platform they prefer.
- For Formatting: Use of forms or templates for submissions.
Editing and Revision
- Tracking Changes – Journals may want to track
revisions to submitted articles, making it easier for reviewers and
editors to see what alterations have been made from the final
“proofs.” Note: This feature appears to be absent in
most software tools.
- User-Friendly interface for reviewers/referees –
As reviewers are already working pro bono, it is crucial that they
find the online reading of articles and submission of reviews
obvious and simple (noted by David Velleman).
Security: Some tools use encrypting technology (SSL) for
document transfer and correspondence.
Summary of articles on scholarly electronic peer
review processes:
Discussions of automated peer review tools for scholarly electronic
publications generally focus on several key issues and
advantages:
- Enhancing speed of submission, communication, viewing and
manipulation of documents among authors, editors and reviewers.
Includes gains of de-centralizing review process and enabling
platform independence via online system.
- Reducing costs of review process, such as those of postage,
printing, and phone.
- Allowing for efficient assignment, tracking and management of
reviews, including automated reminders or enforcement of
deadlines.
- Maintaining records of actions performed and tracking
reviewers’ past performance.
- Archiving data
More general discussions of online peer-review processes focus on
providing a means of filtering and classifying the otherwise
unmanageable scope and quality variation of online content.
Articles:
http://www.consecol.org/Journal/consortium.html
Conservation Ecology Project: Discussion of their own online
submission and review software, currently in use. Provides a
detailed discussion of the review tools and process (includes
double-blind submission and review). Site also includes a
demonstration of the process. See the discussion of upgrade plans
and associated issues and requirements for the software.
Note: They are seeking collaboration with other entities - Contact
Info: “Please contact Shealagh Pope, the Project Coordinator
for the Conservation Ecology Project, at
sepope@consecol.org if you have
any comments or questions”
See also a discussion of software function upgrade plans and
intentions for distribution at
http://www.resalliance.org/consortium/consortium.html
Includes plans for SGML/XML coding of accepted articles.
http://www.press.umich.edu/jep/05-04/edmonds.html
A Proposal for the Establishment of Review Boards BRUCE EDMONDS
Proposal includes graphs of different existing review processes and
a more complex system for automatically assigning material in
specific topic categories to appropriate reviewers.
http://www.firstmonday.dk/issues/issue4_4/proberts/
General discussion of the value of online publication and role of
peer review of scholarly work focusing on time, storage space, and
cost savings.
http://www.press.umich.edu/jep/05-04/sheridan.html
‘Digital Workflow: Managing the Process Electronically’
– Article outlines the features and extols the virtues of
online scholarly publication, including automated peer review
management (“An additional benefit might be that they give
the appearance of reducing bias because they are
systematized”).
http://www.press.umich.edu/jep/03-03/vanreenen.html
Manifesto of sorts advocating library-sponsored alternatives to
commercial publishing. Depicts a battle over electronic
subscription licensing (esp. of scientific and technology
resources) with “predatory” commercial enterprises.
Mentions electronic peer review as an important offering to
potential users. Language conforms with the “learning by
doing” motto of Conservation Ecology site.
http://www5conf.inria.fr/fich_html/papers/P55/Overview.html
"Electronic Management of the Peer Review Process", G.
Jason Mathews and Barry E. Jacobs.
Article from 5th International www conference.
“This paper addresses the electronic peer review process
problem. Namely, how does one electronically manage the complex
process of peer reviewing papers over a physically distributed set
of participants (i.e., authors, reviewers, and
administrators).” - Discusses
technical
dimensions of “Electronic Management System”.
Describes programming tasks for various peer review functions.
Provides examples of interactive forms and requisite code.
http://www.bodley.ox.ac.uk/icsu/donovanppr.htm
"The Truth About Peer Review",
Bernard Donovan
Skeptical discussion of the value of electronic peer review of
scientific journals, focusing on cost savings, effects on academic
community, etc. “In theory, electronic refereeing offers many
benefits, and would be readily accepted by the academic community,
for 63% of the authors canvassed in a study done by the ESPERE
project (Wood,1997)” Article emphasizes the difficulties of
assimilating multiple text and graphics formats, as well as the
uneven technical capacity of users to view and print complex
documents.
http://www.ariadne.ac.uk/issue5/jime/
Article describes an online “argumentation” and review
process for pre-print publications and advocates interactive forms
of online publication. Cites conservation ecology as primary
example of peer review software.
Peer Review Software tools
http://eos.wdcb.rssi.ru/tools/pros.html
Free (for noncommercial use) online review tool
(“pros”). Demo available from this page. Somewhat
limited functionality.
http://www.bepress.com/
PR tool with flexible and extensive submission, assignment and
tracking. Good interface design. Includes an optional
“banking” system whereby authors
accrue“debts” of 3 reviews per submission with a buyout
option (demo set at $1000).
http://peerreview.temple.edu/
“Temple Peer Review Manager” – Developed at
Temple University. Flexible management system for assigning
different editors. Special feature allows blind or open submission
and an “open restricted” correspondence system that
allows authors and reviewers to communicate anonymously. Contact
info: “Please send email questions and comments to Dr. Munir
Mandviwalla at mandviwa@temple.edu or 215-204-8172.”
http://www.consecol.org/Journal/consortium.html
Conservation Ecology Project
See discussion under Articles, above. Software still being
developed and enhanced. Future editions plan to convert documents
to XML, SGML, etc.
http://www.scholarone.com/applications.html
ScholarOne
Commercial submission, editing and peer review software.
Description and demo available. Extensive list of functions.
Submission upload feature promises ‘on-the-fly’
conversion of texts and images (the latter to jpeg format). Contact
info:
Send e-mail to getstarted@ScholarOne.com or call (804) 817-2040
ext. 118.
http://cjs.cadmus.com/rapidreview/index.html
‘Rapid Review’ – Commercial submission and review
tool. Extensive functions. See their list of 24 on their site.
Includes elaborate tracking and report functions, on-line help
system, automatic recognition of user status with different levels
of security, etc. For a personal demonstration of Rapid Review,
please contact Lisande Bissonette.
mailto:bissonettel@cadmus.com
http://www.biomedcentral.com/info/niche.asp
Biomed Central: Central host for biomed journals offering
electronic peer review tools for journals wishing to publish
through them. Central retains ownership of journals published.
http://www.ukoln.ac.uk/services/elib/projects/espere/
ESPERE project: “The experiment is intended to achieve an
introductory level of article submission and peer review by email
transfer of a file which includes all the figures and tables
applicable to the paper.”
http://catalyst.washington.edu/home.html
Catalyst Peer Review Tool: A web-based system for student
collaboration and comments on others’ work. (not a publishing
tool)
Comparative list of features
|
Bepress |
PROS |
Temple |
ConsEco |
Scholar1 |
RapidRev |
|
|
|
|
|
|
|
1. Automated Submission |
X |
X |
X |
X |
X |
X |
2. Automatic Notifications |
X |
|
X |
X |
X |
X |
3. Artilce assignment/tracking |
X |
X |
X |
X |
X |
X |
3.1 Event Logging |
? |
? |
X |
X |
X |
X |
4. Reviewing/Copyediting |
X |
X |
X |
X |
X |
X |
5. Quality/Category Tags |
X |
|
|
|
|
X |
6. Blind/Doubleblind Option |
Blind Auth |
|
X |
X |
? |
X |
7. Time reminders or enforcement |
Flexible |
|
Flexible |
Flexible |
X |
X |
8. Automatic Posting |
X |
|
X |
X |
X |
X |
9. Reviewer Info/Performance |
? |
|
? |
X |
X |
X |
10. Security |
? |
? |
SSL |
? |
X |
Flexible |
Online Journals with electronic
submission/review:
Other Resources
http://www.public.iastate.edu/~CYBERSTACKS/EJI.htm#Accelerated
“A Registry of Innovative E-Journal Features,
Functionalities, and Content” - List of sources on electronic
publishing, including peer review processes.
http://www.espere.org/links.htm
“This page has links to a number of interesting sites who are
active in various aspects of online submission and/or peer
review.”
http://info.lib.uh.edu/sepb/sepb.html
Scholarly Electronic Publishing Bibliography
This selective bibliography presents over
1,350 articles, books, electronic documents, and other sources
that are useful in understanding scholarly electronic
publishing
efforts on the Internet and other networks.
Interview notes
John King
John explained, first of all, that journals should be thought of
primarily as a forum for discussion among academic peers. More
specifically, he provided an extensive analysis of different types
of journals and corresponding priorities for information
“flow.”
Types of Journals:
“Frontwave” – Journals devoted to new science and
evolving discourse.
These journals require limited peer review since authors and their
work are likely already known within a specialized community. The
validity of their experiments and results for the purposes of
publication, he suggests, are relatively easy to determine in this
context.
“Reformulation, Re-thinking” – Primarily social
science and humanities journals that serve the purpose of a public
forum for debate. As their terms are more contested, these journals
generally require more reviews (often three for tie-breaking
purposes).
“Position/Action Journals” – Policy-oriented
journals. [We did not discuss their procedural needs]
Other issues: Quality control, disciplinary boundaries (editors may
need to choose reviewers carefully so as not to elicit a hostile
response based on sub-disciplinary antagonisms that cross a given
topic of study).
Reviews may vary in tone: Those of work by junior faculty can less
afford to be hostile, as it can discouraged them.
John also provided a general outline of review process:
“Case Management:” Basically, the tracking of cases
(articles) through a process with a discrete start and end
(submission and publication). Most issues come down to keeping
track of all cases in process and avoiding lost articles. This
involves reminders and applying pressure to associate editors,
authors (esp. when asked to revise and resubmit) and reviewers (to
release article in a timely manner).
“Workload Management:”
This involve decisions regarding the role of different members of
the journal staff. For example, associate editors may be charged
with different articles and reviews in their respective fields of
expertise.
Authors may pick editors that are most suited to their area and
submit directly to them.
Or, there may be a centralize intake, after which articles are
distributed (John strongly prefers centralized intake so as not to
lose track of articles – the “black hole”
problem, as he put it).
Issues and Problems:
Level of involvement by Editor in Chief: Does the article come back
to the center at every stage in the process (after initial review,
prior to revise/resubmit, etc.)?
Rejection of article by editor regardless of reviews. Sometimes an
editor may wish to be able to over-ride positive (or negative)
reviews.
Electronic correspondence: Issues of version control and
formatting.
Policy regarding access of authors to manuscripts in progress (some
may continue to edit their work indefinitely if given the chance,
he notes).
John discussed the difficulties involved in setting up a system
that allows associate editors to manage different types of articles
separately and then get them all together prior to publication-
suggested the possibility of using a client server database [he was
not sure if this was technically feasible].
David Velleman
David expressed interest in automated review tools, but had several
concerns about compatibility and ease of use.
Compatibility- Philosopher’s imprint asks authors to submit
rtf versions of articles and then allows referees to open them in
whatever format they prefer. Nonetheless, he stressed that it is
important that authors be able to see the formatting and style in
which their article was to be viewed by referees. At present, this
happens in the final stage of the publication process, when final
proofs of articles (as they would appear in print) are sent to
authors as hard copies for them to mark prior to actual
publication. This ability for authors to see the final appearance
of their work and for editors to see their marks on such a copy is
very important for the journal.
Ease of use – David emphasized that the interface for
reviewers in particular must be “obvious,” since their
participation is already a courtesy.
PI uses two reviewers, typically, with a third brought in for
tie-breaking purposes.
David welcomed the idea of a double-blind online conversation
between authors and referees in which each was able to see both
article and review in an online work space. He would like to be
able to do this, but would also like to be able to censor
privileged correspondence between reviewer and editor (himself)
prior to making the review available to the author.