Over the weekend I received a rare honour, a press release directed at me with the full intellectual might of the British Homeopathic Association behind it.

The statement came after I wrote a piece for the Guardian which was published under the title "Homeopathic association misrepresented evidence to MPs". Since they've taken such a personal interest in my work, I feel obliged to respond.

Before I go through their response, it's important to briefly recap the claims I made in that piece. I've highlighted in bold those claims that are simply not addressed at all in the BHA's response.

1. Three of the four systematic reviews cited (Kleijnen, Boissel, Cucherat) did not in fact reach "the qualified conclusion that homeopathy differs from placebo."
2. In fact, Boissel and Cucherat aren't two reviews, but two reports on the same review.
3. The fourth, Linde, was later superseded by a more critical study by the same authors.
4. The presentation of an Ernst review omitted vital caveats.
5. The Jonas paper was not supportive of homeopathy.
6. The BHA have presented a Cochrane review by Vickers as both positive and inconclusive evidence.
7. Vickers is in fact inconclusive, not positive.
8. The BHA cite the need for individualization as an excuse for poor results in trials, but won't explain why the mass produced remedies at Boots are okay.

Note that each of my claims was meticulously linked back to the primary evidence - the papers and statements from authors. This is something the BHA conspicuously fail to do in their release.

Let's work our way through their statement.

The Guardian Science Blog on 5th February has an entry by Martin Robbins that maligns and misrepresents the BHA’s submission to the Commons Science & Technology Committee. We set the record straight by clarifying the facts:

Robbins claims the BHA is wrong to state that four comprehensive reviews of homeopathy "have reached the qualified conclusion that homeopathy differs from placebo". Quoting from the conclusions of each of the four original papers:

There follows an orgy of selective quote-mining.

Kleijnen 1991: “At the moment the evidence of [mainly placebo-controlled] clinical trials is positive but not sufficient to draw definitive conclusions.”

In fact the full quote, which for some reason is edited in the BHA statement, says:

"At the moment the evidence of clinical trials is positive but not sufficient to draw definitive conclusions because most trials are of low methodological quality and because of the unknown role of publication bias."

The conclusion is not "qualified positive" as it was described in the BHA's evidence; it is that any positive evidence that exists is too poor in quality to make a judgement on, and that the results are therefore inconclusive.

"Boissel 1996: “From the available evidence it is likely that among the tested homoeopathic approaches some had an added effect over nothing or placebo.”

Cucherat 2000: “There is some evidence that homeopathic treatments are more effective than placebo.” This paper differs markedly from the report of the same group of authors (Boissel 1996), and indeed contains an update of that review with at least two additional papers’ data cited and analysed.

Again with the blatant quote-mining! The quote from Cucherat in fact reads as follows:

"There is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials. Studies of high methodological quality were more likely to be negative than the lower quality studies."

I spoke to Boissel, author of both reports (which the BHA now admit are essentially the same review with a minor update), and showed him the BHA's interpretation.

His response was crystal clear: "My review did not reached the conclusion 'that homeopathy differs from placebo'." The Boissel review was inconclusive due to poor evidence, it was not a positive conclusion, and in fact it suggests publication bias. Yet the BHA have the gall to attempt to reinterpret Boissel's work even when Boissel himself explicitly states that this is not his conclusion.

Already you can see the pattern. Statements are selectively taken from the literature sometimes cut off mid-sentence so that they can be included in their dossier, minus any akward caveats or clarifications.

Linde 1997: “The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homoeopathy are completely due to placebo”. In 1999, the same group of authors investigated the influence of indicators of methodological quality on study outcome and found that studies with higher-quality scores tended to yield less positive results than those with lower-quality scores. After discarding most of the lower-quality trials, homeopathic treatment remained more effective than placebo – though less strikingly so than in their 1997 analysis. In both of Linde’s systematic reviews, insufficient evidence was found to draw conclusions about the efficacy of homeopathy for any specific medical condition.

I don't disagree with what the BHA have said here, but I'm lost as to how any of the above translates into a "qualified positive" conclusion. The conclusion of Linde's 1999 update was not positive - it was inconclusive, as the BHA's own review shows.

The main thrust of the BHA’s submission to parliament was actually a summary of condition-specific systematic reviews of randomised controlled trials of homeopathy. Robbins states we incorrectly reflect two of those reviews. Again, quoting from the original papers:

Ernst 1999: “There is evidence that homeopathic treatment can reduce the duration of ileus after abdominal or gynecologic surgery. However, several caveats preclude a definitive judgment. These results should form the basis of a randomized controlled trial to resolve the issue.”

Actually this is Barnes & Ernst 1997, and this is absolutely the conclusion that Ernst reached, however this is not how the BHA presented it to parliament. They presented this study with the simple line that it "concluded there was positive evidence for homeopathy." As Ernst himself remarked to me: "They omitted the important caveats from our conclusions and therefore were grossly misleading in the interpretation of our data."

This is a charge to which, notably, the BHA fail to respond.

Jonas 2000: “Overall, it appears that homeopathic remedies work better than placebo in studies of rheumatic syndromes, but there are too few studies to make definitive conclusions about the efficacy of any one type of homeopathic treatment for any one condition.”

So again, why describe this study as "concluding there was positive evidence for homeopathy," when the conclusion was once again inconclusive due to the lack of evidence?

The clinical research evidence in homeopathy is not currently definitive – which is why we need more research – but there are positive findings from randomised controlled trials in a number of medical conditions, some of that evidence supported in systematic reviews.

Vickers 2006: Despite being advised to the contrary (see below), Robbins accuses the BHA of confusion about the Cochrane review of influenza. Depending on the judgmental criteria used, the review may be regarded as positive or inconclusive regarding the treatment of influenza with the homeopathic preparation Oscillococcinum. Vickers concluded that Oscillococcinum reduced the length of influenza illness by 0.28 days on average; the effect was statistically significant, and so the review is included in the BHA website list as a positive finding. For the Committee review, we acknowledged this clearly in a footnote: “The review reported that Oscillococcinum reduced the length of influenza illness by 0.28 days (95% confidence interval, 0.50 to 0.06). The authors concluded “though promising, the data are not strong enough to make a general recommendation to use Oscillococcinum for first-line treatment of influenza”. Robbins has chosen to focus on the Cochrane review’s discussion of influenza prevention: “Current evidence does not support a preventative effect of Oscillococcinum-like homeopathic medicines in influenza and influenza-like syndromes”.

This response is obtuse in the extreme. I'll deal with the misguided claim of statistical significance later, but needless to say, 0.28 days is not significant in the progression of an illness that can only really be measured in days. Can you tell me when your flu finished to within ten minutes, or even six hours?

More interesting is this hand-waving about 'criteria for judgement'. The BHA website is designed for the public, and many lay people won't know the difference between statistical significance (a mathematically notable but not necessarily important result) and a genuinely positive finding. Thus, simply presenting a result like this in a long list of studies with no qualification or explanation is potentially grossly misleading.

More to the point, they talk about these different criteria for the website and for the evidence submission, but what were they? They don't elaborate, and so they haven't actually answered the question the article posed - why was the same review presented as positive on their website, but inconclusive in their evidence check?

And finally, whichever quote you take from the article, the finding isn't positive - it's inconclusive, yet again due to the weakness of the data.

Any reasonable person looking at those reviews would conclude that they are inconclusive, that they show that conclusions about the positive or negative evidence for homeopathy cannot be reached until better quality evidence is available. They would not describe them as four positive results for homeopathy.

The BHA then decide to publish an extract from an e-mail exchange between myself and Robert Mathie. Curiously, they leave out the difficult questions I put to Sumners, and they also omit the follow-up questions I put to Mathie after his response. I didn't have space to deal with all of this material in a 1,000 word Guardian piece, but I'm happy to do so now, as frankly it does the BHA few favours.

A transcript of email dialogue with Mr Robbins on 3rd February illustrates the biased nature of his reporting:

Robbins: Can you clarify what you feel the state of the evidence for homeopathy is? Your spokesman in a recent Telegraph article on 10:23 talked of a growing body of evidence, while your recent press release and statement of evidence to the select committee suggest that the body of evidence is generally poor in quality and quantity.

Mathie: There is research evidence in a number of medical conditions that homeopathic intervention is effective. Our written and oral evidence to the select committee made that clear. To date, a total of only 142 clinical trials in homeopathy have been published in peer-reviewed journals (compared with an estimated half million in conventional medicine), and so of course more research is required.

This seems to me to be a perfectly fair question, and Mathie restates what he told parliament. What I was ideally looking for was some way to tie together what Cristal Sumners had said with Mathie's evidence - in other words for Mathie to show me how the body of evidence had improved in recent years.

Robbins: Why do you think it's been so difficult to produce consistently good results in controlled trials?

Mathie: Actually, 44% of the 142 controlled trials in homeopathy have been positive (see same sources as above). That research is spread rather thinly over 80 different medical conditions, and there are only 28 conditions in which there has been at least two trials. Differing study designs and the small size of many trials means that there are few conditions where there has been an opportunity to achieve consistent results. Nevertheless, repeatably positive results have been obtained for example in hayfever, fibromyalgia, sinusitis and vertigo.

"Actually" implies disagreement, but the 44% figure simply isn't impressive, especially when you take into account the fact that many neutral results were taken as positive. The majority of trials, by Mathie's own admission, have failed to demonstrate the efficacy of homeopathy.

I'm not a PR expert, but even if you accept Mathie's maths, "it works less than half of the time" doesn't strike me as a particular great statement for a press release.

But more notablY, Mathie fails to respond to my actual question. I asked what the state of the evidence was in the previous question, here I was asking why the evidence base remained so poor; why so few studies are being conducted by homeopaths, and why they tend to be of such low quality.

Robbins: Some of the papers included in your evidence seem odd. Just taking the first few for example, you cite a Cochrane review (5,Vickers) as supporting homeopathy, yet the authors very clearly state in the paper (and to me in correspondence) that "Current evidence does not support a preventative effect of Oscillococcinum-like homeopathic medicines in influenza and influenza-like syndromes." I found similar apparent errors through-out this document – how does the BHA explain these anomalies?

Mathie: There is nothing inconsistent or anomalous about the entries in the list. The evidence in systematic reviews is a synthesis of all the research available at the time. We have highlighted the individual trials as well as the systematic reviews. : In the case of influenza, the Cochrane review by Vickers also reported that, when used as treatment, the homeopathic medicine Oscillococcinum reduced the length of influenza illness by 0.28 days on average; the effect was statistically significant, and so the review is included in our list as a positive finding.

We've already covered the errors at length, but let's look at Mathie's comments on the Vickers review.

Mathie's assertion of statistical significance is plain wrong on two counts. Firstly, statistical significance does not necessarily mean a positive result, or any sort of clinical significance. In my follow-up question, I asked if Mathie felt that the general public would understand the difference between a statistically significant result, and a clinically significant result, but he failed to respond.

Secondly, and more fundamentally, 0.28 days is not a statistically significant result. If you're measuring the period of flu, you simply cannot attain this level of accuracy. In fact, assuming that patients were asked how many days their illness lasted, you would expect by random chance an error of around a quarter of a day (you can see a good discussion of this at the Bad Science Forums). (Note, this could be avoided with the use of a sufficiently large sample size, but only 2 studies in Vickers fully meet the inclusion criteria to begin with, leaving their analysis under-powered).

And finally, let us remind ourselves of the conclusions of the authors: "Current evidence does not support a preventative effect of Oscillococcinum-like homeopathic medicines in influenza and influenza-like syndromes."

Vickers et al did not find in favour of homeopathy, and cherry-picking one statistically significant but clinically meaningless result from the paper is not helpful to public understanding.

So that's the end of the BHA's statement. But there's more. Much more. In the Guardian article I concentrated on the systematic reviews, but other trials included are equally dubious. A plethora of randomized controlled trials (RCTs) presented as being "non-conclusive" are emphatically negative. These include:

Hart: "We conclude that arnica in homoeopathic potency had no effect on postoperative recovery in the context of our study."

Stevinson: "Homeopathic arnica was no better than placebo in reducing postoperative complications. These results are compatible with the negative findings from other studies."

Labreque: "CONCLUSION: The homeopathic treatment was no more effective than the placebo treatment of plantar warts."

Kainz: "There was no apparent difference between the effects of homoeopathic therapy and placebo"

Kaziro: "Arnica Montana appeared to give rise to greater pain than placebo (p less than 0.05) and caused more swelling than the placebo (p less than 0.01)."

Yes, homeopathy managed to be worse than placebo, and this was deemed to be, er, inconclusive. Other 'inconclusive' studies fall into a whole other category, for example:

McCarney: A review that brilliant concludes: "Main results: There were no studies that fulfilled the criteria for inclusion and no data to present."

And so on, and on, and on, and you get the idea. Studies that are clearly negative, that clearly concluded that homeopathy was no better than placebo, are counted as 'inconclusive'. It is spectacularly obtuse behaviour.

One claim they didn't respond to is the fact that they were unable to explain whether homeopathic medicines need to be individualized. Since the BHA are releasing correspondence, here's an exchange with Cristal Sumners that they didn't publish:

Robbins: In your statement, you talk about "the many difficulties encountered squeezing a holistic and individualised treatment into a strictly controlled trial methodology," which is absolutely fair enough.

My question is, does this not apply to the mass-produced remedies made by companies like Nelson's, and sold off-the-shelf to millions of customers at Boots? These clearly aren't holistic or individualised treatments, since patients won't receive a personal consultation or assessment. Is it fair to say that in your view, this sort of factory-made treatment really goes against the philosophy of homeopathy?

Would you go even further and say that big pharmaceutical companies like Boots have in some sense corrupted homeopathy for commercial gain, ignoring many of the important aspects of homeopathy that proper homeopathic clinics would use in their treatments? Are big pharmaceutical companies like Boots effectively crowding better and more responsible homeopaths with more holistic approaches out of the market?

Here is the unedited response, in full:

Cristal: The BHA supports a person's legal right to buy and use homeopathic medicines.

Well I tried. But why are the BHA suddenly no reluctanct to explain a very basic fact about how homeopathic works? Either the treatments need to be individualized, in which case Boots' treatments are clearly inferior, or they don't, in which case clearly the excuse for why RCT's tend to fail so often is gone. You can't have it both ways, although I suspect with a sense of tedious inevitability that they will try.

If the BHA expect to retain some credibility, they need to re-examine their scholarship, and be honest with politicians and the public about what the literature they cite actually says. If studies are inconclusive, they should be clearly marked as inconclusive, and if studies are negative, they should not be described as inconclusive. Where conclusions come with caveats, those caveats should be clearly stated.

Because it's very easy, if rather time-consuming, for people like me to check their working.

FOLLOW ME ON TWITTER! @MJROBBINS