School of Computer Science THE UNIVERSITY OF BIRMINGHAM

Why Academic Research Communities Should Switch to Post-Publication Reviewing
(DRAFT: Liable to change)

Aaron Sloman
School of Computer Science, University of Birmingham.
(Philosopher in a School of Computer Science)

Installed: 10 Apr 2011
Last updated: 13 Apr 2011

This paper is http://www.cs.bham.ac.uk/research/projects/cogaff/misc/post-publication-review.html
A PDF version may be added later.

See also this discussion of open-access journals: http://www.cs.bham.ac.uk/research/projects/cogaff/misc/open-access-journals.html

A partial index of discussion notes in this directory is in http://www.cs.bham.ac.uk/research/projects/cogaff/misc/AREADME.html

Background

Following the recent announcement of a new open access journal on a philosophy list, with strong emphasis on "simultaneous policies of blind submissions, double-blind review, and anti-plagiarism" I strongly welcome the decision to make the journal open access (to be added shortly to my list here http://www.cs.bham.ac.uk/research/projects/cogaff/misc/open-access-journals.html) but feel the emphasis on blind reviewing, shared by several other journals and conferences, is misplaced: a bad solution to a real problem.

Below I present some thoughts on an alternative way of addressing the problems, namely using post-publication reviewing. After writing those notes, I thought the idea of post-publication review was so good and so obvious that others must have had the same thought, and sure enough Google quickly responded to a '"post publication" review' query, by pointing me at a number of online papers and discussions, including these (this is just a sample):


Reflections on Blind reviewing vs Post-publication reviewing
DRAFT: Liable to be revised

The announcement of a new open access philosophy journal is excellent news
    http://www.adrianpiper.com/berlinjphil/
but I think the emphasis on blind reviewing may be mistaken.

There are several different reasons.

1. The most important objection in the long term is probably that the original
reasons for only publishing papers after a stringent reviewing process have
gone.

It was once expensive to publish and readers (or their employers) had to cover
the costs through subscriptions. So it was important to filter publications
rigorously so that neither publishers nor readers wasted their money on low
quality papers. The costs of publication on the internet with immediate access
for huge numbers of potential readers are now very much smaller, changing the
arguments.

A premise of the original process was that it is possible for expert reviewers
to assess the quality and long term importance of a research paper. This
premise has repeatedly been challenged in part by showing that different
referees have different judgements of the same paper (I've seen this often with
my own papers) and also by showing that referees can be influenced by knowledge
of who the author is, and other background information, which you have tried to
address.

However, there are many products of human endeavour that are not fully reviewed
until after they have been made public. Those include works of art (paintings,
operas, symphonies, novels, plays, poems), commercial products for consumers
(e.g. cars, washing machines, cameras, television sets, software, dress
materials, clothing, building materials, and many many more), political
manifestos, scientific/academic books, and others.

In view of the technology now available, and the very low cost of publishing on
the internet, I see no reason why post-publication reviewing should not become
the norm with ALL academic/scientific research publications.
(I address some of the problems that could raise, below.)

Moreover, reviews would then be made public, with reviewers named (as they
usually are in the case of published book reviews), and with full right of
reply for the original authors (in the same or another forum).

Moreover, whereas some publications will be relevant only to a small body of
experts, others will have different potential audiences (e.g. students who need
good introductory material, people in other disciplines who need to pick up
good summaries of key work in neighbouring fields, lay readers with an interest
in some research area, students trying to select a research area, etc.). So
there can be different reviews written by different sorts of people for
different sorts of readers and published in different places, whereas now a
research paper is typically reviewed only from a narrow viewpoint associated
with the journal, even if its relevance is potentially much broader.

Post-publication reviewing, with multiple review sites for different sorts of
readers, must be a far better system than one that, as now, tends to evaluate
researchers on the basis of where they publish rather than the quality of what
they publish, and which makes heavy use of citation counts (ignoring the
difference between citers arguing that the work cited had important ideas
that inspired new developments and citers showing up flaws in the cited
work. For example, a large proportion of citations of John Searle's 'chinese
room' paper criticising artificial intellgience are attempts to point out
flaws in his paper -- and many authors re-discover the same flaws, boosting
the citation count of the paper they are criticising.

Public post publication reviewing by well qualified peers, can remove the
weight given to arbitary citations, and high quality published reviews can be
used for assessing researchers rather than the current use of consequences of
secret reviews that determing acceptance or rejection.

If papers are mostly reviewed after publicaton then hiding the author's name,
institution, etc. from reviewers makes no sense.

But I don't think there ever were very good reasons for trying to hide those
details: blind reviewing is a bad solution to a real problem about quality of
reviews. The solution is bad because the solution can itself seriously affect
quality of reviews, and can create unnecessary problems for authors. (Explained
below from the point of view of reviewer and author.)

2. There is no longer any need for publications to be frozen in stone. Many
works of art, software packages, mechanical designs, etc. are improved after
initial launch because the original authors, users, critics, or others have
found flaws and made critical comments.

The same should happen to research papers, instead of what happens now --
authors keep writing new papers with revisions of their previous evidence,
arguments, theories, etc., often published in quite different places, so that a
student or other researcher reading a paper has no easy way of finding out
whether the author has corrected errors, provided new evidence, extended the
theory, etc. That can lead to wasted effort on the part of the reader.

As wikipedia shows, there is now good technology to keep track of changes, so
that readers who want to know when changes were made to a publication, and why,
and by whom, can do so.

When papers that have received a lot of attention are revised, the new versions
will be reviewed (sometimes re-reviewed by reviewers of older versions) and the
reviewers can comment on whether criticisms have been answered adequately,
whether the new theory is better than the old one, whether the revisions are
out of date, because of work done elsewhere, etc. Of course, that sort of
reviewing is incompatible with anonymity.

3. There seems to be a widespread assumption that to ensure that publications
are accepted or rejected exclusively on the basis of their quality a policy of
blind submissions and reviewing is necessary.

I.e. it is assumed that judging academic works exclusively on the basis of
their quality (as opposed to reputation of authors or institutions, or
influence of good or bad personal relations between reviewers and authors,
etc.) is done best by hiding the author's identity.

That assumption may be true in special cases -- e.g. the first publication by a
young author who is the child, or collaborator, or supervisee, of someone very
famous, or someone very infamous.

But my experience as a reviewer suggests that blind reviewing is often a
hindrance to assessment of quality.

That's because very often a paper builds on previous work and it is impossible
to evaluate the latest work well without looking at previous or parallel work,
by the author(s), that provides the context. Someone who is not already an
expert in the field can therefore get a *better* estimate of the quality of a
paper by reading related work by the same author which reveals the bigger
picture of which the paper is a part. For example, it can show that parallel
work in progress, or some previous work, addresses some potential criticisms of
the item being reviewed. I have had some of my submitted publications
criticised for not considering points that I had addressed in earlier papers,
which I could not refer to without revealing authorship.

Another reason why the assumption that blind reviewing supports reviewing on
quality alone, by hiding authorship, is often mistaken is that when a research
community is fairly integrated it is often quite easy to recognize the author
of a paper from the style, the assumptions, the work it claims to be extending
or refuting, etc., especially when it is a paper by a well known, highly
respected or highly controversial researcher.

Moreover, nowadays with search engines it is often not very hard to take some
distinctive phrases from a paper under review and identify closely related work
by the same author, or even a preprint version of the work submitted.

I have done that often, and as a result, the information about the background,
far from reducing the quality or objectivity of my review. has improved it,
including enabling me to write more helpful critical comments to be passed to
the author.

The attempt to make heavy use of blind reviewing expresses the assumption that
most reviewers are unreliable, prejudiced, dishonest, vindictive or have
similar flaws. If that's true the faults in the community will be too bad to be
remedied by blind reviewing and should instead be addressed by mechanism that
identify, expose and criticise people who are guilty of dishonest, biased, or
incompetent reviewing.

Making reviewing a public process with authors having the right to know who has
done the reviewing is a far better remedy than depriving reviewers of relevant
information.

Of course there will still be problems, such as people refusing to review
papers by powerful authors whom they fear. But I think the process of
post-publication reviewing will deal with that: people who have critical
comments on a published paper will be able to express them and there may be
many who are competent to do that well who would not have been asked by the
journal editor before publication.

4. As an author, I have often found that requirements of blind reviewing have
made my task very difficult because my work does not consist of a collection of
separate reports on things done or discovered in isolation: most things I write
extend or revise things I have written previously and trying to present a new
paper without setting the context, which would make it very easy for any
reviewer to identify me (and inspect the papers on my web sites), can be very
difficult.

I have often received reviewers comments that were obviously a result of the
reviewer not knowing anything about the background and making incorrect
assumptions, or failing to see the point. In several cases I believe that being
able to provide an introductory overview of my previous work, with references,
would have produced better reviews.

5. A post-publication review policy would remove the problems of plagiarism
related to reviewing procedures. Reviewers would not have access to the
contents of un-published papers, only already published papers.

(Of course people may have access to unpublished work in other roles, so the
problem of plagiarism will not go away completely.)

6. One of the problems of "self-plagiarism" is a product of the practice of
freezing publications. People who have revised their work are understandably
motivated to publish improved versions, and that can often include re-using
material from the original. Various obstacles to making it clear that the new
paper is a revised version of the old one exist, including journal requirements
for work to be original, etc.

If publications were revisable then I suspect there would be far fewer
publications and much less wasted academic effort. A much higher proportion of
published work would actually be read, instead of  being used mainly for
CV-pumping, as happens now.

Of course, selection and promotion panels would have to change some of their
evaluation procedures. Hopefully this will lead to more decisions being based
on professional judgements of the quality of the work done by applicants rather
than other spurious factors such as quantity, ratings of journals in which
published, citation counts, etc.

I have often seen the current mechanisms, which attempt to use 'objective'
measures lead to the selection of a more solid and dull, but productive,
candidate over a younger much more exciting candidate with clear potential to
achieve deep new results, but who does not yet have a publication track-record.
In bygone days we would have preferred the second applicant. (Appointed on
probation.)

How to control the flood

An obvious problem for any journal offering post-publication reviewing without
pre-publication reviewing is the potential flood of publications. Part of the
answer is that the process may be self-limiting because authors will learn that
publising lots of junk, and lots of repetitions of the same content can lead to
a bad reputation through the review process.

It could also be useful to have two levels of status for published papers:
namely (a) unevaluated and (b) evaluated post-publication by (e.g.) at least
three well recognised reviewers who recommend the paper. Mechanisms for
evaluating reviewers could vary, including using votes by readers of their
reviews or other means. It would not be uncommon for different factions to be
for or against different reviewers. But at least that would be out in the open
and readers could make up their own minds.

A journal could have some not very stringent process for pre-publication
reviews of the unevaluated papers to ensure that their content is relevant to
the journal, the quality of the writing good enough, the presentation
appropriate, and there are no glaring errors that authors need to fix.

Another limitation mechanism could be a small but reasonable publication
charge, to be paid by authors (or their institutions), to help cover the costs
of the journal, e.g. about USD$5.00 per page for the first 5 pages, and a lower
rate for additional pages.

There are likely to be other complications I have not yet thought of but I
suspect none that are fatal to the idea.

I welcome comments and suggestions sent to me by email (serious researchers
will find the address easily). I may add comments to this web site, unless
requested not to.
(I prefer not to use blogging mechanisms, for various reasons.)

Maintained by Aaron Sloman
School of Computer Science
The University of Birmingham