Diana Deca and I edited a collection of 18 visions for open evaluation and the future of scientific publishing. Our Editorial summarising the whole collection is here. The 18 individual papers are here, including my own vision, which is an elaborated and updated representation of the ideas I started to develop on this blog.
The current system
The essential drawbacks of the current system of scientific publishing are all connected to the particular way that peer review is used to evaluate papers. In particular, the current system suffers from a lack of quality and transparency of the peer review process, a lack of availability of evaluative information about papers to the public, and excessive costs incurred by a system, in which private publishers are the administrators of peer review. These problems can all be addressed by open post-publication peer review (OPR). Together with open access (OA), which is generally accepted as desirable, OPR will revolutionize scientific publishing.
Open post-publication peer review (OPR)
Open: Any scientist can instantly publish a peer review on any published paper. The scientist will submit the review to a public repository. Reviews can include written text, figures, and numerical quality ratings. The repository will link each paper to all its reviews, such that that readers are automatically presented with the evaluative meta-information. In addition, the repository allows anyone to rank papers according to a personal objective function computed on the basis of the public reviews and their numerical quality ratings. Peer review is open in both directions: (1) Any scientist can freely submit a review on any paper. (2) Anyone can freely access any review.
Post-publication: Reviews are submitted after publication, because the paper needs to be publicly accessible in order for any scientist to be able to review it. Post-publication reviews can add evaluative information to papers published in the current system (which have already been secretly reviewed before publication). For example, a highly controversial paper appearing in Science may motivate a number of supportive and critical post-publication reviews. The overall evaluation from these public reviews will affect the attention given to the paper by potential readers. The actual text of the reviews may help readers understand and judge the details of the paper.
Peer review: Like the current system of pre-publication evaluation, the new system relies on peer review. For all of its faults, peer review is the best mechanism available for evaluation of scientific papers.
In the future system, peer review is more similar to getting up to comment on a talk presented at a conference. Because these reviews do not decide about publication, they are less affected by politics. Because they are communications to the community, their power depends on how compelling their arguments are to the community. This is in contrast to secret peer review, where uncompelling arguments can prevent publication because editors largely rely on reviewers’ judgments, because there is too little time and no formal mechanism for a judgment of the reviewers’ judgments.
Signed or anonymous: The open peer reviews can be signed or anonymous. Reviews will be digitally authenticated by public-key cryptography. In analyzing the review information to rank papers, signed reviews can be given greater weight if there is evidence that they are more reliable.
Paper selection by arbitrary paper evaluation functions (PEFs): The necessary selection of papers for reading can be based on the reviews and their associated numerical judgments. Any reader can define a PEF based on content and quality criteria and will automatically be informed about papers best conforming to his or her criteria. The PEF could for example, exclude anonymous reviews, exclude certain reviewers, weight evidence for central claims over potential impact of the results etc.
Multiple lenses on the literature
The literature can accessed through webportals that prioritize papers according to different PEFs. Organizations and individuals will define PEFs according to their own priorities. The free definability of PEFs will create a plurality of perspectives of the literature. The continual evolution of multiple PEFs renders the evaluation system “ungamable” because PEFs can be adjusted in response to attempts to game the system.
The nature of a review in the current and future systems
Short answer: Research blogging is important, but we also need a crystallized scientific record of post-publication reviews.
Research blogging fills an important gap: between informal discussions and formal publications. Unlike a private informal discussion, a blog is publicly accessible. Unlike a scientific paper, a blog post can be altered or removed from public access. Blog posts are also often anonymous, whereas papers are signed and author-authenticated.
These more fluid properties of blogs make for their unique contribution to scientific culture. However, the very fluidity of blogs also makes them inadequate as the sole vessel of scientific publishing. In particular, blogging lacks the quality of “scholarly crystallization”.
A scientific publication needs to be crystallized in the sense that it is a constant historical record that can be accessed permanently and therefore cited.
Crystallized scientific publications include papers and reviews. Reviews are crystallized publications that serve mainly to evaluate one or several other crystallized publications. Crystallized publications are typically digitally authenticated documents that reference other scientific publications.
Crystallization does not mean that the work cannot be revised.
Revisions can be made and a revision can “take precedence” over the previous version of a publication. This means the revision will be the first thing seen by the user. However, the author cannot edit a published paper. Instead a revision is a separately published document linked to the previous version of the paper and accompanied by a “justification statement” that addresses the changes (typically in response to reviewer’s comments). The justification statement is needed for the revision to take precedence over the previous version and inherit its references: The authors of signed reviews are automatically informed about revisions and need to either reiterate or revise their supportive or critical reviews. Revisions may also be limited to two per year.
As a consequence, each publication and each revision requires a substantial commitment of its authors. The entire history of original publications and revisions remains permanently publicly accessible and the authors have no right or ability to remove this record.
PLoS ONE is a lower-tier journal. Most scientists have a backlog of potentially highly important papers to catch up on. It is, thus, to be expected that most papers will get few reviews.
If we ever get the ultimate solution, i.e. the ideal free publication system with open post-publication peer review, most papers will get few reviews – just as is the case now for PLoS ONE.
On the one hand, this serves a positive function: The community allocates its resources so as to pay more attention to papers with evidence of high quality. On the other hand, there is the danger that good or great work goes completely ignored.
To avoid the zero-reviews scenario, the traditional editor-soliciting of reviews remains an important mechanism in the new system: It helps get a basic quality estimate for each serious scientific paper, so as to get the broader reception of the paper on its way.
An additional mechanism for the new system is that of author-solicited reviews. This would work similarly to recommendations and could help extremely controversial work to acquire some support. Solicited reviews will need to be marked as author-solicited or editor-solicited, so this information can be taken into account by any automatic paper assessment function.
Open post-publication peer review is distinct from the PLoS ONE system. PLoS ONE combines traditional pre-publication review with post-publication commenting. While it promotes publishing of the pre-publication reviews alongside the paper, it allows reviewers to opt for their reviews to remain secret.
In open post-publication peer review, every review is public, including its numerical ratings. In such a system, each PLoS ONE paper would have two to three editor-solicited reviews at its side (PLoS ONE states that the average number of pre-publication reviews is 2.6) and the papers could be ranked according to their ratings.
In addition, the reviews would assess and rate importance along with technical soundness, whereas in PLoS ONE the reviews assess only technical soundness.
Technical soundness is a low bar, so acceptance in PLoS ONE will not place a random paper high on anyone’s list of reading priorities. The best papers in PLoS ONE probably deserve continued evaluation through peer review. The editor-solited review process should therefore be used to get initial numerical estimates of paper quality on multiple scales including importance. This would allow PLoS ONE fans to prioritize their reading of the journal and comment on the best papers.
Scientists accept requests to review papers in the current system – this will not change. In the current system, scientists are approached by editors and asked to review new papers. They usually comply. In the new system, they will be approached similarly often with the same request – only the reviews will be public.
The motivation to review a paper is greater if the review is an open letter to the community. The fact that reviews are public makes reviewing a more meaningful and motivating activity.
In terms of power, the reviewer loses and gains: The reviewer loses the power to prevent or promote the publication of a paper by means of a secret review. The reviewer gains the power to speak to the whole community about the merits and shortcomings of the paper. The power lost is the kind of power that corrupts, the power gained is the kind of power that challenges us in a positive way.
With power comes responsibility. Again, the reviewer loses and gains: The reviewer loses the responsibility to decide the fate of the paper. This kind of responsibility is an ethical burden and creates political conflicts of interest. (The ethical burden is exacerbated by the fact that the judgment of two to four reviewers often turns out to be mistaken: it takes more minds and time to assess scientific advances.) The reviewer gains the responsibility to contribute to the community’s understanding of the new paper. This kind of responsibility is an honor and part of the collective cognitive process of the scientific community.
Reviewers may choose to post their pre-publication reviews written for the current system. Once an open post-publication peer review system is in place, it exists as an option for scientists. Scientists can post their reviews of papers that appeared in the past, if they think their arguments are still noteworthy and the papers sufficiently important.
Papers will still be submitted to journals and reviewed before publication for some time to come. However, every scientist reviewing a paper will ponder a new option:
After writing this review and submitting it to the journal, I could publish my ideas on this paper alongside the paper itself. So should I write this review as an open letter to the community?
The publication of a subset of the carefully written pre-publication reviews that are currently seen only by editors and authors, will greatly add to the depth of the scientific literature. The reviews chosen by their authors for entry into the open system are likely to be of above-average quality. Unlike secret reviews, open reviews derive their power from compelling argument. This provides a strong scientific (rather than political) incentive.
Reviews will be citable publications in their own right. This will motivate reviewers in terms of quality and quantity. Reviews like papers will be citable publications. Initially reviews will not themselves be peer reviewed publications. As the system develops, however, reviews themselves will receive excitatory and inhibitory references from other publications.
Reviewing will gain status, because it is critical to the hierarchical organization of an exploding body of knowledge. Reviewing will be a more important and more formally acknowledged component of scientific activity than it currently is. This is needed in order to evaluate and integrate our exploding body of knowledge.
Highly visible and controversial papers will generate motivation to publish reviews from the beginning. Controversial studies, such as the upcoming paper on “voodoo correlations” in social neuroscience by Vul et al., already motivate many careful responses. Currently, these responses are published by email and on blogs. Some of them will later appear in journals. If a general system for post-publication review existed, these responses would have been published in that system – in addition to being featured on blogs.
It is interesting to note that the Vul et al. paper has been intensely discussed on the web and covered in Nature, the New Scientist, Scientific American, and Newsweek, but will only “appear” – an almost irrelevant concept – in September 2009. This provides an extreme illustration of one of the shortcomings of the current system of scientific publishing: the substantial publication delays. The journal’s making the paper pretty and physically printing a few copies is really an afterthought in this example. Note, however, that the journal was key in selecting the paper for publication after review, thus justifying the attention it got. The initial three reviews will therefore often be solicited by editors in the new system – as they are in the current system.
The number and content of signed reviews written by a given scientist are public pieces of information, thus motivating scientists to contribute. A general open peer review system allows reviewing activity to be analyzed with the same methods used to analyze other publication activity. For anonymous reviews, each scientist’s number and distribution of ratings should also be made publicly available. This information will enable reviewer-specific rating normalizations to be used in computing an overall value index for a scientific paper. A scientist’s reviewing rate and proportion of signed reviews will be public and might, in extreme cases, affect other scientists’ willingness to review his or her papers.