Open post-publication peer review (brief argument)

The first step toward change is to imagine an alternative. This is a synoptic version of a more extended argument presented here.

The current system of scientific publishing

The current system of scientific publishing is not generally open access, provides only journal prestige as an indication of the quality of new papers, relies on a secretive, intransparent, and inherently noisy paper evaluation process, delays paper publication by many months on average, incurs excessive costs, and is privately and intransparently controlled.

The current system serves two positive functions: (1) It administers peer review and provides an evaluative signal that helps readers choose papers, namely journal prestige. This function is critical to scientific progress. However, the drawbacks listed above suggest that the current system does not serve this function satisfactorily. (2) The current system provides an appealing layout for scientific papers. This function is desirable, but not critical to scientific progress.

Recent positive developments in scientific publishing include the Public Library of Science (PLoS) and other open-access journals, the Frontiers journals, and Faculty of 1000, which provides very brief post-publication recommendations of papers with a simple rating (“Recommended”, “Must-read”, “Exceptional”). ResearchBlogging.org collects blog-based responses to peer-reviewed papers – a big advance as it allows anyone to participate and provide evaluative information. However, as of yet these responses lack numerical ratings, are not digitally signed, and not visible when viewing the target paper itself.

While these developments represent important steps in the right direction, they do not go far enough to fully address all problems related to the way the current system utilizes peer review, namely the lack of quality and transparency of the peer review process, the substantial publication delays, the lack of availability of evaluative information about papers to the public, and the excessive costs incurred by a system, in which private publishers are the sole administrators of peer review.

The crucial innovation: open post-publication peer review

These problems can all be addressed by open post-publication peer review.

Open: Any scientist can instantly publish a peer review on any published paper. The scientist will submit the review to a public repository. Reviews can include written text, figures, and numerical quality ratings on multiple scales. The repository will link each paper to all its reviews, such that readers are automatically presented the evaluative meta-information. In addition, the repository allows anyone to rank papers according to a personal objective function computed on the basis of the public reviews and their numerical quality ratings. Peer review is open in both directions: (1) Any scientist can freely submit a review on any paper. (2) Anyone can freely access any review.

Post-publication: Reviews are submitted after publication, because the paper needs to be publicly accessible in order for any scientist to be able to review it. Post-publication reviews can add evaluative information to papers published in the current system (which have already been secretly reviewed before publication). For example, a highly controversial paper appearing in Science may motivate a number of supportive and critical post-publication reviews. The overall evaluation from these public reviews will affect the attention given to the paper by potential readers. The actual text of the reviews may help readers understand and judge the details of the paper.

Peer review: Like the current system of pre-publication evaluation, the new system relies on peer review. For all of its faults, peer review is the best mechanism available for evaluation of scientific papers. Note however, that public post-publication reviews differ in two crucial respects:

(1) They do not decide about publication – as the papers reviewed are already published.

(2) They are public communications to the community at large, not secret communications to editors and authors.

This makes the peer reviews the equivalent of getting up to comment on a talk presented at a conference. Because these reviews do not decide about publication, they are less affected by politics. Because they are communications to the community, their power depends on how compelling their arguments are to the community. This is in contrast to secret peer review, where uncompelling arguments can prevent publication because editors largely rely on reviewers’ judgments, because there is too little time and no formal mechanism for a judgment of the reviewers’ judgments.

Merging review and reception: Currently review is a time-limited pre-publication process and reception of a paper by the community occurs later and over a much longer period, providing a very delayed – but ultimately important – evaluative signal: the number of citations a paper receives. Open post-publication peer review will remove the artificial and unnecessary separation of review and reception. It will provide for a single integrated process of open-ended reception and review of each paper by the community.

Signed or anonymous reviews: The open peer reviews can be signed or anonymous. In analyzing the review information to rank papers, signed reviews can be given greater weight if there is evidence that they are more reliable.

Digitally authenticated: Reviewers can digitally sign their reviews by public-key cryptography (http://en.wikipedia.org/wiki/Public-key_cryptography). The idea of digitally signed public reviews has been developed here (http://code.google.com/p/gpeerreview/), where further discussion and a basic software tool that implements this function can be found.

Reviews as open letters to the community: Reviews will no longer be secretive communications deciding about publication. They will be open letters to the community with numerical quality ratings that will influence paper visibility on webportals (see below). Open post-publication review will build on the current system by providing a forum for comments and evaluations of papers.

Open posting of private pre-publication reviews: The new system will allow the original pre-publication reviewers of a paper to make their reviews public, so that their work in reviewing the paper can be of benefit to the readers of the paper and to the community at large.

Community control of the critical function of paper evaluation: Open post-publication peer review allows the scientific community to organize the evaluation of papers, thus taking control of this critical function, which is currently administered by publishers.

Improving evaluation quality: The quality of the evaluative signals will be improved by post-publication review for a number of reasons:

(1) Since reviews are open letters to the community, their power is dependent on how compelling they are to the community. (In the present system, a scientist can reject a paper with no good arguments at all – for a high-impact journal perhaps he’ll say that the paper is good, but claim that the finding not sufficiently surprising.)

(2) Many reviews will be signed, so the reviewer’s reputation is on the line: he or she will want to look smart and reasonable. (Anonymous reviews can be down-weighted in assessment functions if they are thought to be less reliable.)

(3) Important papers will accumulate more reviews over time as the review phase is open ended, thus providing an increasingly reliable evaluative signal.

Eventually, journal prestige will no longer be needed as an evaluative signal.

Paper selection by arbitrary evaluation functions: The necessary selection of papers for reading can be based on the reviews and their associated numerical judgments. Any reader can define a paper selection function based on content and quality criteria and will automatically be informed about papers best conforming to his or her criteria. The evaluative function could for example, exclude anonymous reviews, exclude certain reviewers, weight evidence for central claims over potential impact of the results etc.

Webportals as entry points to the literature: Webportals will serve as entry points to the literature, analyzing the numerical judgments in the reviews by different criteria of quality and content (including the use of meta-information about the scientists that submitted the reviews). There will be many competing definitions of quality – a unique one for each webportal or each individual defining his or her own paper evaluation function. Webportals can define such evaluation functions for subcommunities – for scientists too busy (or too lazy) to define their own. A webportal can be established cheaply by individuals or groups whose members share a common set of criteria for paper prioritization.

The ultimate goal: free, instant scientific publishing

Free instant publishing: Once open post-publication peer review provides the critical evaluation function, papers themselves will no longer strictly need journals in order to become part of the scientific literature. They can be published like the reviews: as digitally signed documents that are instantly publicly available. Post-publication review will provide evaluative information for any sufficiently important publication. With post-publication review in place, there is no strong argument for pre-publication review. Publication on the internet can, thus, be instant and reviews will follow as part of the integrated post-publication process of reception and evaluation.

Peer-to-peer editor choice: After publication, the author asks a senior scientist in his or her field to edit the paper. If the senior scientist accepts, an acknowledgment of his or her role as editor will be added to the paper. The editor’s job is to select two to four reviewers and to email them with the request to publicly review the paper. If they decline, the editor has to find replacements.

However, anyone else is allowed to review the paper as well. In particular, the author may also inform other scientists of the publication and ask them to review the paper. Author- and editor-requested reviews will be marked as such. Requested as well as unrequested reviews can be signed or unsigned.

Editors must not have been at the same institution or on any paper with the authors. Reciprocal or within-clique review editing is monitored and discouraged. Such information, will in any case be publicly available after the fact and may be used to weight the reviews in any automatic quality assessment.

Revisions: If the weight of the criticism in the accumulated reviews and the importance of the paper justify it, the authors have the option to revise their paper. The revision will then be the first thing the reader sees when accessing the paper and the authors’ response to the reviews may render the criticism obsolete. However, the history of revisions of the paper, starting with the originally published version, will always be available – along with the complete history of reviews and author responses.

Reviewing-activity statistics: Reviewers must be registered as professional scientists whether they sign a given review or not. The level of reviewing activity of a scientist is a public piece of information. Similarly, it is public information, how many reviews were written anonymously by each scientist. (However, it is not public information, of course, which papers a scientist reviewed anonymously.) In general, each scientist is expected to write about as many reviews as he or she receives.

The process of reception and review: Good papers will accumulate many and positive judgments over time and bubble up in the process – some after 4 reviews and 2 weeks past publication, others after years.

High-prestige publications such as Nature and Science could take note of independently published studies that have turned out to be important. Based on the broader and more reliable evidence of public review, they could decide to showcase, i.e. to republish, these projects – perhaps in a modified version suited for their more general audience. These high-prestige journals would, thus, benefit from the greater quality of the broader and deeper public evaluation of the papers, which would contribute to the quality of their product.

Advertisements

11 Responses to Open post-publication peer review (brief argument)

  1. Chris Said says:

    Nikolaus – Great post. I really think you hit the nail on the head identifying the problems with the current state of publishing. I wonder if you could discuss any ideas about how to really incentivize post-publication reviews. Looking at PLoS One, most of the comments sections are empty. One option, which you mentioned, is to have people expect that they should write as many reviews as they receive. But how do you think this could be enforced? I don’t think social norms will be enough.

  2. futureofscipub says:

    chris, this is a crucial question indeed. i’ve posted my ideas on the answer here:
    https://futureofscipub.wordpress.com/questions-and-answers/
    –niko

  3. […] Brief argument for open post-publication peer review […]

  4. If only more than 71 people would read this..

  5. more than 71 people says:

    Very nice indeed. Could this be happening soon?
    However, I am very concerned about “anonymous reviewers”. There should be an upper bound for the number of anonymous reviewers. Anonymity should also have a lifetime (say, one up to five years) which is chosen by the “anonymous reviewer”, where afterwards, his PKU signature will be shown to the public. You see, I am totally against being anonymous.

    -Rawi

  6. James says:

    http://scholarlykitchen.sspnet.org/2011/03/28/how-the-internet-changed-medical-journals/

    Nice to hear your talk at scibarcamb, thought you might appreciate this post from a great blog on (scientific) publishing

    James

  7. […] different approach to post-publication peer review lets the traditional journals continue their work, but then adds another layer of evaluation. This […]

  8. […] different approach to post-publication peer review lets the traditional journals continue their work, but then adds another layer of evaluation. This […]

  9. from artist says:

    from artist…

    […]Open post-publication peer review (brief argument) « The future of scientific publishing[…]…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: