Saturday, February 4, 2012

Stop deifying "peer review" of journal publications:

Peer review.  It is a critical part of scientific research and scientific progress.  Without it, science as a field might look like Fox News Stories or postings on Jenny McCarthy's web site, where ideas people have are given gravitas regardless of how ludicrous they are.  But somehow, many in the public and press, and many many scientists alas, have deep misconceptions about peer review.

The most recent example of such misconceptions involves the arsenic life saga.  If you are not familiar with this story - here is a summary (for some fine scale details on the early parts of the story see Carl Zimmer's post here).

In November 2010 NASA announced that in a few days they would hold a press conference discussing a major finding about life in the universe.  On December 2, 2010, they held their press conference and discussed a paper that was in press in Science from multiple NASA funded authors including Felisa Wolfe-Simon.  The paper was of interest because it claimed to have shown that a bacterium was able to replace phosphate in its macromolecules, including its DNA, with arsenic.  The press conference made claims that were very grandiose, like that textbooks would have to be rewritten, and the study of life on Earth and elsewhere would have to be completely rethought.

After a few days of mostly very glowing press reports, a few critiques began to emerge including in particular one from Rosie Redfield, a microbiologist at the University of British Columbia.  The critiques then snowballed and snowballed and the general consensus of comments appeared to be that the paper had fundamental flaws.  Some of the critiques got way too personal in my opinion and I begged everyone to focus on the science not personal critiques.  This seemed to work a little bit and we could focus on the science, which still seemed to be dubious.  And many, including myself, expressed the opinion that the claims made by the authors in the paper and by the authors and NASA in the press conference and in comments to the press, were misleading at best.

Now critiques about new findings are not unusual.  We will get back to that in a minute.  But what was astonishing to me and many others, was how NASA and the authors responded.  They said things like:
... we hope to see this work published in a peer-reviewed journal, as this is how science best proceeds.
and
It is one thing for scientists to “argue” collegially in the public media about diverse details of established notions, their own opinions, policy matters related to health/environment/science. 
But when the scientists involved in a research finding published in scientific journal use the media to debate the questions or comments of others, they have crossed a sacred boundary [via Carl Zimmer]
and the kicker for me was a letter Zimmer posted
Mr. Zimmer, 
I am aware that Dr. Ronald Oremland has replied to your inquiry. I am in full and complete agreement with Dr. Oremland’s position (and the content of his statements) and suggest that you honor the way scientific work must be conducted. 
Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated. You can see many examples in the journals Science and Nature, the former being where our paper was published. This is a common practice not new to the scientific community. The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner. 
Regards,
Felisa
This was amazing since, well, they were the ones who held the overhyped press conference.  And then I (and others) found it appalling that they in essence would not response to critiques because they were not "peer reviewed." I told Zimmer
Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.
Zimmer had a strong defense of scientists "discussing" the paper:
Of course, as I and others have reported, the authors of the new paper claim that all this is entirely inappropriate. They say this conversation should all be limited to peer-reviewed journals. I don’t agree. These were all on-the-record comments from experts who read the paper, which I solicited for a news article. So they’re legit in every sense of the word. Who knows–they might even help inform peer-reviewed science that comes out later on.
(I note - yes I am quoting a lot from Zimmer's articles on the matter and there are dozens if not hundreds of others - apologies to those out there who I am not referencing - will try to dig in and add other references later if possible).

And so the saga continued.  Rosie Redfield began to do experiments to test some of the work reported in the paper.  Many critiques of the original paper were published.  The actual paper finally came out.  And many went about their daily lives (I keep thinking of the Lord of the Rings whisper "History became legend. Legend became myth. And for two and a half thousand years, the ring passed out of all knowledge."  Alas, the arsenic story did not go away.

And now skipping over about a year.  The arsenic story came back into our consciousness thanks to the continued work of Rosie Redfield.  And amazingly and sadly, Wolfe-Simon's response to Rosie's work included a claim that they never said that arsenic was incorporate into the bacterium's DNA.  (I have posted a detailed refutation of this new "not in DNA" comment here).

But that is not what I am writing about here.  What is also sad to me are the continued statements by the paper's authors that they will not discuss any critiques or work of others unless they are published in a peer reviewed article.

For example, see Elizabeth Pannisi's article in Science:
But Wolfe-Simon and her colleagues say the work on arsenic-based life is just beginning. They told ScienceInsider that they will not comment on the details of Redfield's work until it has been peer reviewed and published.
So - enough of an introduction.  What is it I wanted to write about peer review?  What I want to discuss here is that the deification of a particular kind of journal peer review by the arsenic-life authors is alas not unique.  There are many who seem to have similar feelings (e.g., see this defense of the Wolfe-Simon position).  I believe this attitude towards peer review is bad for science.  Fortunately, many others agree (e.g., see this rebuttal of the defense mentioned above) and there is a growing trend to expand the concepts of what peer review is and what it means (see for example, David Dobbs great post about peer review and open science from yesterday).

Though much has been written about peer review already (e.g., see Peer review discussion at Nature as one example), I would like to add my two cents now - focusing on the exalted status some give to peer reviewed journal articles.  I have three main concerns with this attitude which can be summarized as follows
  1. Peer review is not magic
  2. Peer review is not binary
  3. Peer review is not static.
I suppose I could stop here but I should explain.

Regarding #1 "Peer review is not magic.". 
What I mean by this is that peer review is not something that one can just ask for and "poof" it happens.  Peer review of articles (or any other type of peer review for that matter) frequently does not work as sold - work that is poor can get published and work that is sound can get rejected.  While it may pain scientists to say this (and brings up fears of FoxNews abusing findings) it is alas true.  It is not surprising however given the way articles get reviewed.

In summary this is how the process works.  People write a paper.  They then submit it to a journal. An editor or editors at the journal decide whether or not to even have it reviewed.  If they decide "no" the paper is "sent back" to the authors and then they are free to send it somewhere else.  If they decide "yes" to review it, the editors then ask a small number of "peers" to review the article (the number usually ranges from 2-3 in my field).  Peers then send in comments to the editor(s) and the editor(s) then make a "decision" and relay that decision to the authors.  They may say the paper is rejected.  Or they may say it is accepted.  Or they may say "If you address the comments of the reviewers, we would consider accepting it".  And then the authors can make some revisions and send it back to the editors.  Then it is reviewed again (sometimes just by the editors, sometimes by "peers").  And it may be accepted or rejected or sent back for more revisions.  And so on.

In many cases, the review by peers is insightful, detailed, useful and in the best interests of scientific progress.  But in many cases the review is flawed.  People miss mistakes.  People are busy and skim over parts of the paper.  People have grudges and hide behind anonymity.  People can be overly nice in review if the paper is from friends.  People may not understand some of the details but may not let the editors know.  Plus - the editors are not completely objective in most cases either.  Editors want "high profile" papers in many cases.  They want novelty.  They want attention.  This may lead them to ignore possible flaws in a paper in exchange for the promise that it holds.  Editors also have friends and enemies.  And so on.  In the end, the "peer review" that is being exalted by many is at best the potentially biased opinion of a couple of people.  At worst, it is a steaming pile of ... Or, in other words, peer review is imperfect.  Now, I am not saying it is completely useless, as peer review of journal articles can be very helpful in many ways.  But it should be put in its rightful place.

Regarding #2: "Peer review is not binary"
The thumbs up / thumbs down style of peer review of many journal articles is a major flaw.  Sure - it would be nice if we could apply such a binary metric.  And this would make discussing science with the press and the public so much easier "No ma'am, I am sorry but that claim did not pass peer review so I cannot discuss it" "Yes sir, they proved that because their work cleared peer review."  But in reality, papers are not "good" or "bad".  They have good parts and bad parts and everything in between.  Peer review or articles should be viewed as a sliding scale and not a "yes" vs. "no."

Regarding #3: "Peer review is not static"
This is perhaps the most important issue to me in peer review of scientific work.  Peer review of journal articles (as envisioned by many) is a one time event.  Once you get the thumbs up - you are through the gate and all is good forever more.  But that is just inane. Peer review should be - and in fact with most scientists is - continuous.  It should happen before, during and after the "peer review" that happens for a publication.  Peer review happens at conferences - in hallways - in lab meetings - on the phone - on skype - on twitter - at arXiv - in the shower - in classes - in letters - and so on.  Scientific findings need to be constantly evaluated - tested - reworked - critiqued - written about - discussed - blogged - tweeted - taught - made into art - presented to the public - turned inside out - and so on.

Summary:
In the end - what people should understand about peer review is that though it is not perfect, it can be done well.  And the key to doing it well is to view it as a continuous, nuanced activity and not a binary, one time event.  
UPDATE 3: some twitter comments



Posted by at
Labels: , ,