Peer Review

I have had enough with the peer-review process, both as a reviewer and as an author. If you Google “alternatives to peer review,” you will see that I am not alone. There were many ideas proposed and even implemented to mitigate the pitfalls of this outdated approach. My goal here is not to analyze these methods, but only to rant about my experience with peer-review ineffectiveness in the world of GNSS.

 

I must have reviewed over 60 manuscripts in the last 5 years. With each review requiring between 2 and 10 hours, I think I invested a significant amount of time and energy in improving the quality of scientific publications. While my standards as a reviewer were originally quite high, they gradually lowered as I reviewed countless low-quality articles. I initially convinced myself that my constructive criticisms were in fact an educational experience for students. Still, I couldn’t really understand why supervisors could not offer better guidance, and why they even accepted to put their names on such manuscripts. I eventually understood that quantity gained precedence over quality, and perhaps for a reason: a long list of references helps researchers get more credentials, increases their chances to get referenced, all of which leads to better funding opportunities. Publish or perish might describe the academic world.

 

My most unpleasant experience as a reviewer occurred when I received a revised manuscript in which the authors had simply ignored almost all comments and suggestions I had devoted hours to come up with. Authors might not necessarily agree with all comments made by reviewers, but they should at least show some respect and acknowledge the work that went into the review. It is a clear insult to do otherwise. Editors also have more and more trouble finding reviewers for papers, and it is easily understandable since nobody wants to waste precious time on non-innovative, poorly-written manuscripts from arrogant authors.

 

My experience as an author of peer-reviewed papers is more limited: I typically refuse to publish material more than once, even though it does not always get the exposure it deserves at conferences. Still, I have had my share of frustrations. One reviewer recommended outright rejection of what I consider to be my most innovative paper… thankfully, the other two reviewers weren’t as cynical and counterbalanced the decision by suggesting only minor revisions. I realize that I have at times been a ruthless reviewer myself, but I believe that I always gave proper credit to work that truly deserved it.

 

Another disappointing experience occurred recently when the review process for my manuscript was subject to unexpected delays. Coincidentally, another paper with similar content was submitted to the same journal shortly before my review came back. Even though I can’t discard the fact that these authors may have come up with the same innovative idea as mine with strangely inadequate timing, I can’t discard either the possibility that reviewers (and most likely researchers from their network of connections) shared, used and even re-published material that was under review. This is truly unethical and is almost impossible to condemn with an anonymous review process.

 

The peer-review process used to be an opportunity to learn about innovative algorithms, provide input to improve scientific methods and receive useful feedback to fine-tune our own work. Unfortunately, it seems to get harder and harder to get real benefits from this process, and I will certainly stay tuned on the latest developments in this area. Meanwhile, I still have to decide what to do with my latest draft...



Write a comment

Comments: 3
  • #1

    Jim Ray (Monday, 16 May 2016 11:25)

    Simon, I share most of your same concerns. The entire review process has broken down and no longer serves it purpose adequately. In addition to your points, I would add these:
    * Allowing reviews to be anonymous creates needless opportunities for conflicts of interest and ethical failures. I will not review for any journal that fails to respect non-anonymous reviewing.
    * University PhD requirements for minimum numbers of refereed papers have greatly increased the reviewing burden (especially in Germany and to a lesser extent in China), unnecessarily in my view.
    * I too have seen major reviewing efforts dismissed with a simple "beyond the scope of the paper" or similar. So I try to point out to the editor which comments absolutely should not be ignored. But ultimately the problem can only be adjudicated by the editor, which requires they be sufficiently well informed. Sadly, that is often not the case, even at some leading journals.
    * The opposite problem, of editors giving undeserved weight to some bogus reviews, also can only be addressed by having informed editors. No review with unsupported or overly broad claims should be respected in the face of conflicting views. But I recently saw a totally unsupported (and incorrect!) one-sentence review to overrule three extended but contrary reviews.
    * Some journals are far too incestuous in allowing associate editors to ease the publication of allied papers while suppressing competing views. This is highly unethical but almost never regulated, probably because of the increasing difficulty of finding new editors.

    It is not clear to me how major improvements can be made, but these steps would help: 1) ban anonymous reviews; 2) identify reviewers and editors in the acknowledgements of all papers; 3) allow reviewers/editors to attach short statements to publications (e.g., in cases where the authors ignore reviewer comments); 4) make the entire process fully transparent; 5) create processes to allow ethical complaints to be submitted; 6) pursue and prosecute ethical violations more vigorously.

    Thanks for creating this blog!

  • #2

    Simon Banville (Tuesday, 17 May 2016 21:08)

    Thanks Jim for sharing additional insights! Actually, in an effort to improve the transparency of the process, I used to include my name in the reviews. I then realized that this initiative could dangerously backfire, since not everybody appreciates constructive criticisms. Hence, I am concerned that a non-anonymous peer-review process could create tensions between researchers.

    I am currently leaning towards self-publishing: do we really need papers to be peer-reviewed or can we let the public decide on the quality of the work? ResearchGate seems to be moving in this direction by generating DOIs for unpublished work, allowing for comments on publications, and of course by providing a searchable archive of papers. The concept may still need refinements but I believe it is a good framework to start with.

  • #3

    Jim Ray (Wednesday, 18 May 2016 08:05)

    Given the pathetic quality of many of today's journal papers, I agree that self-publication would at least bypass ethical roadblocks and uninformed editors. But it would be better if operators of such services also published comments (signed) by readers. Authors could then make revisions, if they agreed and after a certain period the paper would be finalized. I believe the EGU has at least one open journal of this sort.