Q: How can scientists be motivated to submit reviews in an open peer review system?

Scientists accept requests to review papers in the current system – this will not change. In the current system, scientists are approached by editors and asked to review new papers. They usually comply. In the new system, they will be approached similarly often with the same request – only the reviews will be public.

The motivation to review a paper is greater if the review is an open letter to the community. The fact that reviews are public makes reviewing a more meaningful and motivating activity.

In terms of power, the reviewer loses and gains: The reviewer loses the power to prevent or promote the publication of a paper by means of a secret review. The reviewer gains the power to speak to the whole community about the merits and shortcomings of the paper. The power lost is the kind of power that corrupts, the power gained is the kind of power that challenges us in a positive way.

With power comes responsibility. Again, the reviewer loses and gains: The reviewer loses the responsibility to decide the fate of the paper. This kind of responsibility is an ethical burden and creates political conflicts of interest. (The ethical burden is exacerbated by the fact that the judgment of two to four reviewers often turns out to be mistaken: it takes more minds and time to assess scientific advances.) The reviewer gains the responsibility to contribute to the community’s understanding of the new paper. This kind of responsibility is an honor and part of the collective cognitive process of the scientific community.

Reviewers may choose to post their pre-publication reviews written for the current system. Once an open post-publication peer review system is in place, it exists as an option for scientists. Scientists can post their reviews of papers that appeared in the past, if they think their arguments are still noteworthy and the papers sufficiently important.

Papers will still be submitted to journals and reviewed before publication for some time to come. However, every scientist reviewing a paper will ponder a new option:

After writing this review and submitting it to the journal, I could publish my ideas on this paper alongside the paper itself. So should I write this review as an open letter to the community?

The publication of a subset of the carefully written pre-publication reviews that are currently seen only by editors and authors, will greatly add to the depth of the scientific literature. The reviews chosen by their authors for entry into the open system are likely to be of above-average quality. Unlike secret reviews, open reviews derive their power from compelling argument. This provides a strong scientific (rather than political) incentive.

Reviews will be citable publications in their own right. This will motivate reviewers in terms of quality and quantity. Reviews like papers will be citable publications. Initially reviews will not themselves be peer reviewed publications. As the system develops, however, reviews themselves will receive excitatory and inhibitory references from other publications.

Reviewing will gain status, because it is critical to the hierarchical organization of an exploding body of knowledge. Reviewing will be a more important and more formally acknowledged component of scientific activity than it currently is. This is needed in order to evaluate and integrate our exploding body of knowledge.

Highly visible and controversial papers will generate motivation to publish reviews from the beginning. Controversial studies, such as the upcoming paper on “voodoo correlations” in social neuroscience by Vul et al., already motivate many careful responses. Currently, these responses are published by email and on blogs. Some of them will later appear in journals. If a general system for post-publication review existed, these responses would have been published in that system – in addition to being featured on blogs.

It is interesting to note that the Vul et al. paper has been intensely discussed on the web and covered in Nature, the New Scientist, Scientific American, and Newsweek, but will only “appear” – an almost irrelevant concept – in September 2009. This provides an extreme illustration of one of the shortcomings of the current system of scientific publishing: the substantial publication delays. The journal’s making the paper pretty and physically printing a few copies is really an afterthought in this example. Note, however, that the journal was key in selecting the paper for publication after review, thus justifying the attention it got. The initial three reviews will therefore often be solicited by editors in the new system – as they are in the current system.

The number and content of signed reviews written by a given scientist are public pieces of information, thus motivating scientists to contribute. A general open peer review system allows reviewing activity to be analyzed with the same methods used to analyze other publication activity. For anonymous reviews, each scientist’s number and distribution of ratings should also be made publicly available. This information will enable reviewer-specific rating normalizations to be used in computing an overall value index for a scientific paper. A scientist’s reviewing rate and proportion of signed reviews will be public and might, in extreme cases, affect other scientists’ willingness to review his or her papers.