Getting a comment published  in PRL is far from trivial. I recently succeeded in publishing my Comment on: How the result of a single coin toss can turn out to be 100 heads.  Despite the fact that I pointed out a major  error in a published paper, the road to publication was long and difficult.  The odds were against me (5 other comments on the same paper were rejected despite being correct) and I had to fight the authors of the original paper on top of a biased editor and a referee.   I hope this post will help anyone  thinking about writing a comment (or a reply to a comment).  This was my second attempt at a PRL comment, the first one did not get published as a comment but eventually became a very well received paper.

I attempted to be as general as possible and minimize the specific technical details of  my comment. For anyone interested in the content, the arXiv version is slightly more complete.  It contains a one paragraph response to the published reply. This short response is a very good summary of the comment.

Publishing a comment in PRL

1. A PRL with errors

So you think PRL XXX has mistakes and makes false claims, and you think you should let the world know. What better way than submitting a comment in PRL?  Well… You’re in for a treat.

Generally PRL will only publish a comment if it identifies a central error. This has  following implications: If the paper in question is completely meaningless it cannot contain an error and is therefore comment-proof. If the discussion of the results is speculative to the point where it is not supported by the results of the paper, it is also safe, unless you can convince the editor that the discussion is a central point. etc..

Example 1: The paper I commented on made absurd claims such as:

Our results provide evidence that weak values are not inherently quantum but rather a purely statistical feature of pre- and postselection with disturbance

such claims, although unjustified, are unfortunately safe from comments since it is almost impossible to demonstrate they are the central point of the paper.

My guess is that the majority of the five or six comments submitted on this specific paper were rejected for this reason. Each comment (e.g 1,2,3,4) showed that central features of weak measurements were missing in the supposed `classical analogue’. The relation to weak values is so weak that all subsequent conclusions about the nature of weak values are speculative at best.  But wild speculation is not a central mistake and the claim is safe from criticism in PRL.

Technical note: I commented on a paper that supposedly provided a classical analogue for anomalous weak values. The central result of the paper was a measurement scheme that provides strange results which they claimed are classical weak values. Most of the other comments showed that the scheme is not a weak measurement in one way or another. My comment was different in that I simply showed that they are using a nonsensical method to calculate the result of the measurement.

2. You will probably end up fighting with both the referee and authors of the original paper

Before submitting your comment, you may want to contact the authors of the original paper to see their reaction. I guess that in some cases scientists will put science ahead of their ego, but in many (most?) cases they will not. The result is that they will try to fight the comment. You should be aware of this.

The editor also has an ego. He accepted the paper and he does not like to admit he accepted a paper with a major mistake (remember PRL only accepts comments that point out central errors).

After you submit the comment, the editor will usually send it directly to the original authors (the other option is a quick reject). They will then write a report claiming that your comment is not worth publishing.

Here you are at a disadvantage. The editor will probably side with the authors. The likely outcome is a reject or (as in my case) a “we cannot accept” unless you make a good rebuttal.

Example 2: A few years ago we tried to write a comment on a paper titled “Vanishing Quantum Discord is Necessary and Sufficient for Completely Positive Maps“. We provided a counterexample to the statement in the title. The main objections of the authors were that we were working under different assumptions. WTF? There were no explicit assumptions in the paper (or in the preceding literature) that contradicted our counterexample, moreover they never event attempted to point out what the different assumptions were. Nevertheless the editor agreed with the authors. The counterexample never made it as a comment (despite going through phase 3-4 below), but did eventually turn into a very nice paper.

3. The rebuttal

If things went well, this is the point where the editor asks for a rebuttal of the claims in the informal reply. Now, here is the most important piece of advice. Do not make any changes to your comment at this stage! Just as the authors try to show that your comment is a piece of junk, you should at this point show that their reply is a joke. Remember, the authors are not your friends, they are not trying to improve your comment, they want it out of their way.

So you need to plan ahead. When submitting the comment you must anticipate all possible replies by the authors. If your argument is correct, and furthermore if you hit a mistake that may be considered critical, the rebuttal should not be a major task. This is what I learned from my previous experience and it paid off.

One issue that appeared in all author responses (to this comment and the previous one) was misdirection. On top of the attempt to show that the claims are not a central issue, the authors try to misdirect the editor/referee and move the argument in a different direction.  The best advice is to try and ignore those issues that are tangential to your central claim.

Example 3:  In both the informal and published reply, the authors claimed that there are other papers with classical analogues of weak measurements. The issue has little to do with my comment. My comment was: There is a mathematical error in this paper. I don’t really care if some other guys have similar results. I never responded to that criticism.

4. If you were successful with your rebuttal, your paper will be sent to a referee

The referee will give you comments on your manuscript. Treat them like any other referee reports. In my case they made very good points and asked for clarifications and changes before making a decision. I made the changes and the paper is much much better as a result.

It is perhaps important to remember that the referee will get some of the editor’s bias, in-fact in my previous comment we got the following negative report “I agree with the editor”.

5. Back to the authors

And again they get to reply, and again they can make it as long as they want.

I was reasonably lucky at this stage because the authors are not experts in the field and it was very easy to point out mistakes in their reply. As before, I did not make any changes based on the author’s reply, and only responded to real criticism.

The paper went back to the referee and he made some suggestions on possible changes but otherwise recommended publication.

6. The reply

At this point the authors had to give their one page reply. This reply did not go to referees. It was sent to me, but the editor explained that it will be published regardless of what I say. My only reply was to point out two technical errors. The authors quoted some results that do not exist and are plain wrong (in fact they base their argument on an inequality where the units don’t match).  The editor was nice enough to let me add a (very brief) note to the comment pointing out these errors. He was also nice to the original paper’s authors and allowed them to make the same type of change (but nothing else), i.e add a note at the end of their paper. Their note contained another mistake (actually the same mistake again) and subsequently they now have a reply with three major technical errors.

This is a lesson for anyone writing a reply. It might not be refereed so make fucking sure it is all correct. One way to get some feedback is to use similar arguments to the ones used to fight the comment. If you make up new things at the last moment you are risking it.

Since there was no option to react to their reply, I added a short response on the arXiv version. Actually this short response is better than my comment and it is a shame it would not get published, but such is life.

7. Have fun

If like me, you like a good argument, a comment is an extreme challenge with a referee to help decide the winner. At the end of the day (actually months) the back and forth can be enjoyable. On top having to make concise and precise arguments I  had to read the literature presented in the counter-arguments, this forced me to read some  nice papers that I had been putting off for a while, or missed.  Moreover, each time  it was great to learn how well I anticipated the  attempts at refuting my comment (I did not anticipate the  mathematical mistakes that appeared in  the published reply).

 

8. Conclusion

Writing a comment can be a very rewarding experience, especially if it gets published. On the other hand, comments are a lot more work than you would expect. The process is ugly and biased against the person making a comment. As a rule, PRL editor try to fight off comments and make to road to publication tough. The upside is that getting it published is extremely gratifying.