Homeopathy and the Absence of Evidence
Posted by apgaylard on April 26, 2008
The apologists for homeopathy are upset again: this time with Professor Edzard Ernst and Simon Singh’s new book. Why? Because once more homeopathy is exposed for what it is: a placebo.
Dr Damien Downing, the Medical Director for the Alliance for Natural Health (ANH), seems particularly put out. So much so that he has released a rather silly critique. (This link seems to be down. Try here)
After some empty carping he suggests that Ernst is not a very good scientist and then goes on to wrap himself in the flag of good science, “The scientific method ‘consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses’ (Wikipedia) – not of unsubstantiated dogmatic statements. Science has no room for dogma.”
This is one point I can agree with; compared to the statements of some other protagonists it’s pretty reasonable: science should have no room for dogma. However, Downing is not averse to peddling some homeopathic propaganda.
This is clearly seen in the way he handles evidence. He refers to, what I would expect, is his best scientific evidence base for homeopathy: the homeopathy evidence section of “The National Library for Health“. He points out that it currently, in his view, “contains 32 systematic reviews and meta‐analyses of [homeopathy’s] use in a wide range of disorders“. He opines, “Of the 32, 7 report a statistically significant clinical effect from homeopathy, 6 show a non‐significant trend in its favour, and 3 show no effect; 16 concluded that there was “insufficient data” to draw a conclusion either way.”
Now I’ve very closely examined this database in the past and could not disagree more strongly with this ‘analysis’. It completely misses the main point: if you are concerned with either treating patients, or recovering from illness, what you really need to know is how many of these reports show that homeopathy works as well as the recommended conventional treatment: the answer is none.
[three more items of ‘evidence’ have been added since I looked at the database, none of these have, as yet, been through the complete review process so it would be premature to cite these – hence my review is still relevant.]
Next, the idea of a “statistically significant clinical effect” needs some thought. Note that it does not claim that there is a clinically significant effect. That is, an effect that would be worth having. This is good, because none of the reports show any.
A statistically significant effect just means that the statistical tests used on the data indicate that if the study was repeated it is likely that the difference in outcomes between groups given homeopathic sugar pills and ordinary sugar pills would be seen again. In other words, the difference between these two groups is not likely to be just the result of chance. However, not likely here usually means nineteen times out of twenty. Not too reassuring when you see the number of outcomes measured in some of these studies, or the number of researchers in the world looking at this issue.
Neither do tests of statistical significance account for a range of other impediments including experimenter bias, publication bias, lack of blinding, high drop-out rates from control groups etc.
One of the silliest things in Downing’s statement is the citing of studies that he contends show “a non‐significant trend” in favour of homeopathy. If the trend is not statistically significant, then it’s very likely to be nothing more than random noise in the experiment. This is non-evidence, not evidence.
So we have no evidence to support the use of homeopathy; the authors of these ‘positive’ studies actually ask for more research. All we have is a small number that claim to reach statistical significance for the particular ‘homeopathic’ intervention showing an effect in excess of a placebo.
The next sleight of hand is the contention that, “16 concluded that there was “insufficient data” to draw a conclusion either way.” The “insufficient data” part seems to be presented as a quotation. This phrase does not appear in sixteen of the reports.
Also, the observation that researchers have looked for an effect and not found one tells its own story. Whilst, in an absolute sense absence of evidence is not evidence of absence; this is too simplistic. We need to remember to account for prior probability. If something genuinely doesn’t exist then, by definition, we are never going to find evidence of its existence: there is an absence of evidence of unicorns because they are a myth. Similarly, absence of evidence for the deeply physically implausible practise of homeopathy is telling us something.
However, if we are looking for something that is likely to be real, the quality of the search for evidence is also important: absence of evidence in high-quality research is clearly informative in a way that a similar result in low-quality research is not.
Seen in this light one of the observations contained in this database, that Downing has mysteriously overlooked, is vital:
“… Studies of high methodological quality were more likely to be negative than the lower quality studies …” [Cucherat et al]
In other words, the better the quality of the search for a homeopathic effect the less likely one is to be found!
This moves us onto the oddly vexatious topic of the famous review by Shang et al. For Downing, “…the authors identified 110 relevant studies and then excluded all but 8 of them from the final analysis – and declined to name them! This would seem to be blatant research misconduct.”
Reality is somewhat different. The research progressively excluded studies on the basis of transparent quality criteria. They were particularly interested in bias and found that is correlated strongly with sample size. That only eight of over one hundred trials of homeopathy made the cut tells us something important about the quality of research conducted into homeopathy. That the eight best studies, taken together, showed that homeopathy is no more than a placebo is an entirely proper conclusion – consistent with the findings of Cucherat et al. Any plea to include more of the original 110 is a plea for the inclusion of bias: not good science. [this subject is excellently explored on Paul Wilson’s blog]
Again, the better the quality of the search the more negative the findings about homeopathy. This is, of course, what would be expected if there were no benefits from homeopathic remedies (aside from the placebo effect): seemingly positive results are just noise in the signal and can be removed by proper filtering.
The assertion that the authors refused to name the final eight studies is a persistent piece of homeomythology. I have commented on this before at some length. The truth is that Shang and his co-authors unwisely omitted the names of the eight studies from the original paper; some people pointed this out and they named them in the 17th December 2005 issue of The Lancet. They have also made the details of the included and excluded papers available on a website. This all happened in 2005! It would seem to me that to raise the banner of good science requires that one, at the very least, keeps up to date with developments!
On the subject of Shang et al, Downing confuses proper scientific conduct with mis-conduct. This egregious folly can only be the result of a shocking lack of competence or letting personal dogma cloud his judgement. I prefer to think it is the latter. In any event, given this woeful performance, it would seem rather embarrassing to vilify Ernst as a bad scientist.
It’s worth noting that Downing’s much vaunted “National Library for Health” database contains one review that, taken at face value, is very problematic for a homeopathy advocate.
A meta-analysis of homeopathy for postoperative ileus by Barnes et al was not able to reach a definitive judgement. However their data indicated that studies working with potencies below 12C (there could be some active agent left) provided a statistically significant reduction in time to first flatus (vs. placebo) whereas those using potencies above 12C (odds are that just the solvent is left) did not. Now, because homeopathic ‘remedies’ are usually diluted to potencies beyond 12C it both flatly contradicts both usual homeopathic practise and the ‘less is more’ notion of the ‘law’ of infinitesimals.
Finally, the most positive review contained in this database, covering trails of a homeopathic ‘medicine’ for vertigo (that many homeopaths wouldn’t recognise as proper homeopathy anyway!) made this plea:
“… The positive effects of Vertigoheel in vertigo are based on good levels of evidence, but larger trials are required …” [Karkos et al]
If this is the best evidence that apologists for homeopathy have to offer one wonders why they bother. The real answer is that this debate is not about the evidence at all; it is about some believers in an out-moded quasi-religious system of medicine clutching at fig-leaves to cover their embarrassment.
Science certainly has no room for dogma. Propaganda isn’t that helpful either, but that is all Downing and other apologists are peddling.
19 Responses to “Homeopathy and the Absence of Evidence”
Sorry, the comment form is closed at this time.