Getting a comment published in PRL is far from trivial. I recently succeeded in publishing my Comment on: How the result of a single coin toss can turn out to be 100 heads. Despite the fact that I pointed out a major error in a published paper, the road to publication was long and difficult. The odds were against me (5 other comments on the same paper were rejected despite being correct) and I had to fight the authors of the original paper on top of a biased editor and a referee. I hope this post will help anyone thinking about writing a comment (or a reply to a comment). This was my second attempt at a PRL comment, the first one did not get published as a comment but eventually became a very well received paper.
I attempted to be as general as possible and minimize the specific technical details of my comment. For anyone interested in the content, the arXiv version is slightly more complete. It contains a one paragraph response to the published reply. This short response is a very good summary of the comment.
Publishing a comment in PRL
1. A PRL with errors
So you think PRL XXX has mistakes and makes false claims, and you think you should let the world know. What better way than submitting a comment in PRL? Well… You’re in for a treat.
Generally PRL will only publish a comment if it identifies a central error. This has following implications: If the paper in question is completely meaningless it cannot contain an error and is therefore comment-proof. If the discussion of the results is speculative to the point where it is not supported by the results of the paper, it is also safe, unless you can convince the editor that the discussion is a central point. etc..
Example 1: The paper I commented on made absurd claims such as:
Our results provide evidence that weak values are not inherently quantum but rather a purely statistical feature of pre- and postselection with disturbance
such claims, although unjustified, are unfortunately safe from comments since it is almost impossible to demonstrate they are the central point of the paper.
My guess is that the majority of the five or six comments submitted on this specific paper were rejected for this reason. Each comment (e.g 1,2,3,4) showed that central features of weak measurements were missing in the supposed `classical analogue’. The relation to weak values is so weak that all subsequent conclusions about the nature of weak values are speculative at best. But wild speculation is not a central mistake and the claim is safe from criticism in PRL.
Technical note: I commented on a paper that supposedly provided a classical analogue for anomalous weak values. The central result of the paper was a measurement scheme that provides strange results which they claimed are classical weak values. Most of the other comments showed that the scheme is not a weak measurement in one way or another. My comment was different in that I simply showed that they are using a nonsensical method to calculate the result of the measurement.
2. You will probably end up fighting with both the referee and authors of the original paper
Before submitting your comment, you may want to contact the authors of the original paper to see their reaction. I guess that in some cases scientists will put science ahead of their ego, but in many (most?) cases they will not. The result is that they will try to fight the comment. You should be aware of this.
The editor also has an ego. He accepted the paper and he does not like to admit he accepted a paper with a major mistake (remember PRL only accepts comments that point out central errors).
After you submit the comment, the editor will usually send it directly to the original authors (the other option is a quick reject). They will then write a report claiming that your comment is not worth publishing.
Here you are at a disadvantage. The editor will probably side with the authors. The likely outcome is a reject or (as in my case) a “we cannot accept” unless you make a good rebuttal.
Example 2: A few years ago we tried to write a comment on a paper titled “Vanishing Quantum Discord is Necessary and Sufficient for Completely Positive Maps“. We provided a counterexample to the statement in the title. The main objections of the authors were that we were working under different assumptions. WTF? There were no explicit assumptions in the paper (or in the preceding literature) that contradicted our counterexample, moreover they never event attempted to point out what the different assumptions were. Nevertheless the editor agreed with the authors. The counterexample never made it as a comment (despite going through phase 3-4 below), but did eventually turn into a very nice paper.
3. The rebuttal
If things went well, this is the point where the editor asks for a rebuttal of the claims in the informal reply. Now, here is the most important piece of advice. Do not make any changes to your comment at this stage! Just as the authors try to show that your comment is a piece of junk, you should at this point show that their reply is a joke. Remember, the authors are not your friends, they are not trying to improve your comment, they want it out of their way.
So you need to plan ahead. When submitting the comment you must anticipate all possible replies by the authors. If your argument is correct, and furthermore if you hit a mistake that may be considered critical, the rebuttal should not be a major task. This is what I learned from my previous experience and it paid off.
One issue that appeared in all author responses (to this comment and the previous one) was misdirection. On top of the attempt to show that the claims are not a central issue, the authors try to misdirect the editor/referee and move the argument in a different direction. The best advice is to try and ignore those issues that are tangential to your central claim.
Example 3: In both the informal and published reply, the authors claimed that there are other papers with classical analogues of weak measurements. The issue has little to do with my comment. My comment was: There is a mathematical error in this paper. I don’t really care if some other guys have similar results. I never responded to that criticism.
4. If you were successful with your rebuttal, your paper will be sent to a referee
The referee will give you comments on your manuscript. Treat them like any other referee reports. In my case they made very good points and asked for clarifications and changes before making a decision. I made the changes and the paper is much much better as a result.
It is perhaps important to remember that the referee will get some of the editor’s bias, in-fact in my previous comment we got the following negative report “I agree with the editor”.
5. Back to the authors
And again they get to reply, and again they can make it as long as they want.
I was reasonably lucky at this stage because the authors are not experts in the field and it was very easy to point out mistakes in their reply. As before, I did not make any changes based on the author’s reply, and only responded to real criticism.
The paper went back to the referee and he made some suggestions on possible changes but otherwise recommended publication.
6. The reply
At this point the authors had to give their one page reply. This reply did not go to referees. It was sent to me, but the editor explained that it will be published regardless of what I say. My only reply was to point out two technical errors. The authors quoted some results that do not exist and are plain wrong (in fact they base their argument on an inequality where the units don’t match). The editor was nice enough to let me add a (very brief) note to the comment pointing out these errors. He was also nice to the original paper’s authors and allowed them to make the same type of change (but nothing else), i.e add a note at the end of their paper. Their note contained another mistake (actually the same mistake again) and subsequently they now have a reply with three major technical errors.
This is a lesson for anyone writing a reply. It might not be refereed so make fucking sure it is all correct. One way to get some feedback is to use similar arguments to the ones used to fight the comment. If you make up new things at the last moment you are risking it.
Since there was no option to react to their reply, I added a short response on the arXiv version. Actually this short response is better than my comment and it is a shame it would not get published, but such is life.
7. Have fun
If like me, you like a good argument, a comment is an extreme challenge with a referee to help decide the winner. At the end of the day (actually months) the back and forth can be enjoyable. On top having to make concise and precise arguments I had to read the literature presented in the counter-arguments, this forced me to read some nice papers that I had been putting off for a while, or missed. Moreover, each time it was great to learn how well I anticipated the attempts at refuting my comment (I did not anticipate the mathematical mistakes that appeared in the published reply).
Writing a comment can be a very rewarding experience, especially if it gets published. On the other hand, comments are a lot more work than you would expect. The process is ugly and biased against the person making a comment. As a rule, PRL editor try to fight off comments and make to road to publication tough. The upside is that getting it published is extremely gratifying.
I like the first paragraph of the second page of the arxiv version.
It reminds me of trouble I and a colleague had with a paper that was not yet (but in the end was) published in PRL. After getting nowhere in dialogue with the authors, we wrote our paper which said (along with much else) something along the lines of “If one accepted Hoban et al’s definitions, then the completely ridiculous conclusion X would follow”. A referee for PRL obviously asked them to address our arxiv paper. They added to their paper something like “As shown by Rieffel and Wiseman […] our results can also be applied to show X.” Aaaargh.
When it comes to discord these things happen all too often. I have three papers that suggest discord should not be identified as a resource for (gate based) QC. Surprisingly they are cited in sentences of the type “Discord is also a resource for quantum computation …. “.
A nice and somewhat relevant blog post on responding to reviewers.
Are there other options? Could one publish the comment paper in a completely different journal? Then you’d avoid all the problems you talk about.
Maybe that depends on how important the paper you’re attacking is, or how interesting your paper is as a standalone???
Indeed this is a possibility. The downside, is that it does not get immediately associated with the paper you are commenting on. In the case of PRL, the page for the paper contains a link to the comment, so it is easier to discover. My first experience trying to get a comment to PRL ended up as a PRA paper, but did require a lot of back and forth with the PRA editor since it was considered controversial. At the end, it worked out well as a paper, probably better than it would have as a comment.
How did you know 5 other comment papers were rejected? Did the editor tell you, or did you find & contact the authors of those papers? Where did those papers end up being published?
I know most of the authors of the other papers as well as the authors of the paper I commented on. Also, as far as I could tell, all comments were sent to the same referee who did mention the other comments in the first report. I also found some information on the other comments from Facebook.
Regarding publication, I know that Vaidman’s comment was eventually turned into a paper https://arxiv.org/abs/1703.08870 . I don’t think there were any follow ups for the others. Also, there were other people who went directly to writing a paper, e.g. https://arxiv.org/abs/1410.0943.
This is relevant to me. A group from a well-known company with a lot of money has done work that has had huge media publicity. The thing they produced seems good, but their testing is very flaky and the written paper very sloppy. e.g. Building a quantum communication system or crypto system that probably is very good, but claiming it beats the best classical system, has better performance, scaling, etc. based on bad testing that is degrading the classical system’s performance, etc. The hype has made people draw wild conclusions about both their system & quantum comm. or quantum crypto. (This is an example only.) Only a small fraction of their data is shown & it looks like they won’t release any more ever, even though there’s no reason for them not to. (The data would not help rival researchers.) This work may turn up in one of those high-publicity publications, maybe the one beginning with N.
There’s some online criticism, but it’s been overwhelmed by the hype & people don’t seem to have noticed big flaws. I’m going to write a paper about this, but of course don’t have access to the product or their full data. Right now, I’m not sure exactly where to submit it, but I guess to a more specialized journal. The paper will be like a critique, a lot different from what I’m used to. It feels like it’s going to be like an extended version of what the reviewers should be saying but won’t.
Can any reader of this comment add any suggestions?