Q: How can scientists be motivated to submit reviews in an open peer review system?

Scientists accept requests to review papers in the current system – this will not change. In the current system, scientists are approached by editors and asked to review new papers. They usually comply. In the new system, they will be approached similarly often with the same request – only the reviews will be public.

The motivation to review a paper is greater if the review is an open letter to the community. The fact that reviews are public makes reviewing a more meaningful and motivating activity.

In terms of power, the reviewer loses and gains: The reviewer loses the power to prevent or promote the publication of a paper by means of a secret review. The reviewer gains the power to speak to the whole community about the merits and shortcomings of the paper. The power lost is the kind of power that corrupts, the power gained is the kind of power that challenges us in a positive way.

With power comes responsibility. Again, the reviewer loses and gains: The reviewer loses the responsibility to decide the fate of the paper. This kind of responsibility is an ethical burden and creates political conflicts of interest. (The ethical burden is exacerbated by the fact that the judgment of two to four reviewers often turns out to be mistaken: it takes more minds and time to assess scientific advances.) The reviewer gains the responsibility to contribute to the community’s understanding of the new paper. This kind of responsibility is an honor and part of the collective cognitive process of the scientific community.

Reviewers may choose to post their pre-publication reviews written for the current system. Once an open post-publication peer review system is in place, it exists as an option for scientists. Scientists can post their reviews of papers that appeared in the past, if they think their arguments are still noteworthy and the papers sufficiently important.

Papers will still be submitted to journals and reviewed before publication for some time to come. However, every scientist reviewing a paper will ponder a new option:

After writing this review and submitting it to the journal, I could publish my ideas on this paper alongside the paper itself. So should I write this review as an open letter to the community?

The publication of a subset of the carefully written pre-publication reviews that are currently seen only by editors and authors, will greatly add to the depth of the scientific literature. The reviews chosen by their authors for entry into the open system are likely to be of above-average quality. Unlike secret reviews, open reviews derive their power from compelling argument. This provides a strong scientific (rather than political) incentive.

Reviews will be citable publications in their own right. This will motivate reviewers in terms of quality and quantity. Reviews like papers will be citable publications. Initially reviews will not themselves be peer reviewed publications. As the system develops, however, reviews themselves will receive excitatory and inhibitory references from other publications.

Reviewing will gain status, because it is critical to the hierarchical organization of an exploding body of knowledge. Reviewing will be a more important and more formally acknowledged component of scientific activity than it currently is. This is needed in order to evaluate and integrate our exploding body of knowledge.

Highly visible and controversial papers will generate motivation to publish reviews from the beginning. Controversial studies, such as the upcoming paper on “voodoo correlations” in social neuroscience by Vul et al., already motivate many careful responses. Currently, these responses are published by email and on blogs. Some of them will later appear in journals. If a general system for post-publication review existed, these responses would have been published in that system – in addition to being featured on blogs.

It is interesting to note that the Vul et al. paper has been intensely discussed on the web and covered in Nature, the New Scientist, Scientific American, and Newsweek, but will only “appear” – an almost irrelevant concept – in September 2009. This provides an extreme illustration of one of the shortcomings of the current system of scientific publishing: the substantial publication delays. The journal’s making the paper pretty and physically printing a few copies is really an afterthought in this example. Note, however, that the journal was key in selecting the paper for publication after review, thus justifying the attention it got. The initial three reviews will therefore often be solicited by editors in the new system – as they are in the current system.

The number and content of signed reviews written by a given scientist are public pieces of information, thus motivating scientists to contribute. A general open peer review system allows reviewing activity to be analyzed with the same methods used to analyze other publication activity. For anonymous reviews, each scientist’s number and distribution of ratings should also be made publicly available. This information will enable reviewer-specific rating normalizations to be used in computing an overall value index for a scientific paper. A scientist’s reviewing rate and proportion of signed reviews will be public and might, in extreme cases, affect other scientists’ willingness to review his or her papers.

4 Responses to Q: How can scientists be motivated to submit reviews in an open peer review system?

  1. Teammacep says:

    Maaaan, you know there is such thing in the web like search engine, http://google.com if you don’t, go there to understand why this post is bullshit

  2. […] peer review model, someone like this would have even read my paper. There are some good arguments here about how reviewers could be motivated to submit reviews in an open peer review system, but I worry […]

  3. Tristan Nakagawa says:

    How about politics? In the end, researchers are driven by funds and grants. Money is a great motivator, and (thorough) reviewing takes time. So reviewing should not be a cost to the scientist, but an investment. If we get grant policy makers to stepwise implement obligations for everybody involved in projects to review 1 (later 2, 3, …) papers on Post-publishing peer review platforms, This perceived cost would be part of the obligations that come along with getting money from grants. Reviews could automatically get articles assigned to them by the platform based on research area, with possibility (and public record) of rejecting to review a certain article and to get assigned a new one.

    Personally, I think the pure ‘reputation’ motivation mentioned is a good point, but obligations/incentives to review would help massively to rapidly make this reality

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: