Quacking About Ducks

In the furore that has followed Ben Goldacre‘s trenchant deconstruction of homeopathy several homeopathic apologists have stepped forward.

Denis MacEoin, a novelist and a former lecturer in Islamic studies, writing in the Guardian’s “Comment is Free“ section ably demonstrated that you get what you pay for.  

It is interesting, in passing, to reflect that MacEoin styled himself a “sceptic” with sympathies for a “pro-science” stance when, according to his wiki entry he has been “…An advocate of alternative medicine since the 1960s… For many years, until its demise in 2003, he was chairman, then president of the Natural Medicines Society …”

Anyway, what really caught my eye was a comment on this piece by the well known US homeopath Dana Ullman.  He, among other things, asked an interesting question:

” … Why do skeptics ignore the fact that three large double-blind placebo controlled trials were conducted on patients with influenza, all showing that a homeopathic medicine called Oscillococcinum was effective. Even the highly respected Cochrane Report acknowledged that these results were “promising.” …”

It is gratifying to note that Ullman refers to the “… highly respected Cochrane Report …”.  It is good to see a homeopath acknowledge the worth of this evidence based medicine resource.  It seems only fair to test his comment against what that “… highly respected …” document contains.

First a simple point of clarification, at least some sceptics are quite happy to engage with this evidence (see here and here, for example).  In fact, this ‘evidence’ has already been debated with Ullman at some length.  So it’s just not true that those of a sceptical disposition “ignore” this information.  It would seem we engage with it frequently.

This is, perhaps, because this ‘evidence’ just keeps cropping up.  It’s also a favourite of Peter Fisher, the Queen’s homeopath.  In an article, published as part of this year’s Homeopathy National Knowledge Week, entitled: “Is homeopathy useful in infectious conditions?” he cited it in some detail (emphasis mine):

“… Influenza has been the subject of the largest number of RCTs of homeopathy of any infectious condition and a Cochrane Review of homoeopathic Oscillococcinum and similar homeopathic preparations for preventing and treating influenza and influenza-like syndromes has been published. Oscillococcinum is made from wild duck heart and liver, and wild fowl are vectors of influenza viruses. The review  found three prevention trials (n= 2265) and four treatment trials (n = 1194). There was no evidence that homoeopathic treatment can prevent influenza-like syndrome, but Oscillococcinum treatment reduced the length of influenza illness by 0.28 days and increased the chances that a patient considered treatment to be effective (RR 1.08; 95% CI 1.17 to 1.00). The authors concluded that the data were promising, but not strong enough to make a general recommendation to use Oscillococcinum for influenza and influenza-like syndromes. Further research is warranted. “

We can also contrast Ullman’s use of the evidence with this statement.

The three prevention trails actually contain nearly twice the number of patients as the four treatment trials.  The result: no evidence that homeopathy can prevent influenza-like syndrome.

Here we can address part of Ullman’s question.  There were not “three large double-blind placebo controlled trials …” , there were four relatively small ones (n_ave =  298.5).  He must be getting confused with the three larger prophylaxis trials (n_ave =  755).

The four treatment trials are listed in Table 1.

Study	Participants (n)
Casanova 1984	100
Casanova 1992	300
Ferley 1989	487
Papp 1998	372

Table 1.  The Treatment Trials and Their Sizes

In fact, the Cochrane Review authors estimated that confirmatory trials would require sample sizes of 2,000 to 50,000 to detect what they called the “very moderate benefit” of Oscillococcinum; depending on the outcome of interest.

This puts the description of these trials as “large” into some perspective.

Were these trials “… conducted on patients with influenza …” as Ullman would have it?  No; Fisher calls the ailments “…influenza and influenza-like syndrome …” with good reason.

The Cochrane Review notes that the treatment studies recruited “Patients … presenting with influenza or influenza-like syndromes (symptoms of influenza, such as cough, fever, chills and muscle pain, in the absence of laboratory evidence of infection).”

It also made the following comments

• “… In two of the four treatment trials, patients had to meet a defined standard for influenza-like syndrome (for example, rectal temperature more than 38 ^oCelsius and at least two episodes of headache, stiffness, lumbar and articular pain or shivers) …” So in two they did not.

• “… Only one of the treatment studies (Ferley 1989) explicitly reported that patients were accrued during an outbreak of influenza …”

So, it is not clear how many of the patients actually had influenza.  It may appear to be nit-picking, but given the vehemence of Ullman’s contribution he cannot really expect any quarter.

Anyway, back to Fisher’s more temperate piece.  What are we to make of the comment that “…Oscillococcinum treatment reduced the length of influenza illness by 0.28 days …”? The Cochrane Review includes data on statistical significance: “…Oscillococcinum treatment reduced the length of influenza illness by 0.28 days (95% CI 0.50 to 0.06).”

Well, a reduction of 0.28 days in the duration of the illness seems very modest.  This average of 6 hours 43 minutes is statistically significant (at the 5% level), but is this really practically significant?  It’s worth noting that the Cochrane Review comments: “This effect might be as large as half a day and as small as about an hour.”

What about increasing “… the chances that a patient considered treatment to be effective (RR 1.08; 95% CI 1.17 to 1.00) …”?  This is a very small treatment effect, based on the patient’s subjective impression of their symptoms.   Looking at the Confidence Intervals we can actually say that this result is not statistically significant at the 5% level.  Why?

Here’s a very clear explanation from the Link Mobile Telecommunications and Health Research Programme (MTHR) Report (2007), (emphasis my own):

“… The effect that chance can play on the odds ration obtained in a study is usually expressed by the 95% confidence interval (CI), which is the range of odds ratios consistent with the data allowing for statistical variations.  The 95% CI indicates that, if 100 similar studies were carried out, the odds ratio found in 95 would be expected to lie within the CI, while in 5 they would lie outside by chance.  In general, if the CI includes 1.0, the results are said not to be significantly different from 1.0, ie not statistically significant: no convincing evidence of a statistical association …”

The CI quoted by Fisher includes 1.0; we have a statistically insignificant result at the 5% level.

It’s also worth considering this comment from Ioannidis:

“… The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. Power is also related to the effect size. Thus research findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3-20), than in scientific fields where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1-1.5) …  In the same line of thinking, if the true effect sizes are very small in a scientific field, this field is likely to be plagued by almost ubiquitous false positive claims. For example, if the majority of true genetic or nutritional determinants of complex diseases confer relative risks less than 1.05, genetic or nutritional epidemiology would be largely utopian endeavors …”

This review is dealing with quite small effect sizes and is thus likely to be suffering from the tendency to false positive claims Ioannidis describes.

Both Ullman and Fisher referred to the Cochrane Review.  What did that conclude?  Its “plain language summary” states: “Homoeopathic Oscillococcinum does not prevent influenza but might shorten the length of the illness … Trials do not show that homoeopathic Oscillococcinum can prevent influenza. However, taking homoeopathic Oscillococcinum once you have influenza might shorten the illness, but more research is needed.“

So, as we have noted: no prophylaxis, but maybe a small shortening of the illness’ duration.

Ullman contends that the medication was found to be effective.  It was not.  The patient’s slightly positive view of the preparation’s effectiveness is not statistically significant.  If it was effective the authors would have recommended its use.

Fisher’s more honest appraisal of the review notes that “… The authors concluded that the data were promising, but not strong enough to make a general recommendation to use Oscillococcinum for influenza and influenza-like syndromes. Further research is warranted.”

There are other important details in the Cochrane Review.  These, no doubt, underlie the author’s cautious conclusions.

“… The standard of trial reporting was poor. For only two studies was there sufficient information to complete data extraction fully (Ferley 1989; Papp 1998) …”

One of the treatment trials (Casanova 1992) was unpublished; the other (Casanova 1984) “… was reported in a general medical magazine rather than in a scientific journal.”

The Review also found the sample sizes in the two Casanova papers to be “… suspiciously round numbers (100 and 300) …”

So, in summary, we can answer Ullman’s question as follows.

1. We don’t ignore the data.  Many of us are happy to engage with it.  He knows this.

2. Four relatively small studies were conducted.  Two were by the same author.  One of these was unpublished, the other published in a general medical magazine.  Both had, what the Cochrane Review called “…suspiciously round numbers …” of participants.

3. It’s not clear that all the patients had influenza.  In the treatment trials, these were “Patients … presenting with influenza or influenza-like syndromes …”

4. The homeopathic medicine was not effective.  The Review states that “… taking homoeopathic Oscillococcinum once you have influenza might shorten the illness, but more research is needed.”  The slight positive outcome for the patient’s view of effectiveness was not statistically significant.

5. The results were called “promising“, but under “implications for practice” the Review stated that “… the evidence was not strong enough to make a general recommendation to use Oscillococcinum for routine treatment …”

Fisher’s use of the data is clearly more considered and honest.  However, it is disappointing that he quotes a statistically insignificant result without comment.

Summary Tables For The Treatment Trials Extracted From The Cochrane Review.

Study Casanova 1984   Methods Treatment allocation: B Performance bias: B Observer blinding: B Exclusions/withdrawals: B   Participants 100 patients with influenza-like syndrome onset less then 48 hours previously. No details of method of recruitment or exclusion criteria. Average age verum/placebo: 42/41 years. Male:female in verum/placebo: 19:31/26:24   Interventions Oscillococcinum, four doses in over two days at six-hour intervals   Outcomes Patient global assessment of success; presence of chills, aches rhinitis, night cough, day cough, fever at day eight   Notes Reported in what appears to be a general medical magazine: very few experimental details given   Allocation concealment B – Unclear

 

Study Casanova 1992   Methods Treatment allocation: B Performance bias: B Observer blinding: B Exclusions/withdrawals: B   Participants 300 patients complaining of influenza. No details of inclusion or exclusion criteria. Average age verum/placebo: 44/38. Male:female in verum/placebo: 61:89/56:94   Interventions Oscillococcinum, twice a day for three to four days   Outcomes Temperature recorded twice a day for four days (data for evening of second day used for continuous outcome, data for evening of day four converted to binomial outcome of fever, by using normal distribution); presence of chills, aches at day four   Notes Inconsistency between text and table: the table appears to have been printed the wrong way around. The text value was selected   Allocation concealment B – Unclear

 

Study Ferley 1989   Methods Treatment allocation: A Performance bias: A Observer blinding: A Exclusions/withdrawals: A   Participants 487 patients presenting in primary care with a complaint of influenza-like syndrome. Inclusion criteria: age older than 12 years; rectal temperature above 38 Celsius and at least two of headache, stiffness, lumbar and articular pain, shivers. Exclusion criteria: duration more than 24 hours; immune deficiency; local infection; immunisation against influenza; depression; immunostimulant treatment. Average age verum/placebo: 34/35. Male:female in verum/placebo: 93:127/97:129   Interventions Oscillococcinum, twice a day for five doses   Outcomes Patient assessment of success; recovery at 48 hours (defined as rectal temperature below 37.5 Celsius and complete resolution of all five symptoms); number of days to recovery; number of days to return to work; use of medication for pain or fever; use of medication for cough or sore throat; use of antibiotic medication; patient assessment of success   Notes Use of medication calculated from percentages given in text. Some minor inconsistencies between figures suggest a small amount of missing data   Allocation concealment A – Adequate

Study Papp 1998   Methods Treatment allocation: A Performance bias: A Observer blinding: A Exclusions/withdrawals: A   Participants 372 patients recruited in primary care or by internal medicine specialists. Inclusion criteria: rectal temperature above 38 Celsius; muscle pain or headache; one of shivering, cough, spinal pain, nasal irritation, malaise, thoracic pain, periarticular pain. Exclusion criteria: duration more than 24 hours; immune deficiency; local infection; immunisation against influenza; medical need for medication; immunostimulant or immunosuppressive treatment. Use of analgesics, antibiotics or anti-influenza agents in the first 48 hours was a post-randomisation exclusion criterion. Average age verum/placebo: 35/35. Male:female in verum/placebo: 95:93/96:88   Interventions Oscillococcinum three times a day for three days   Outcomes Complete recovery after 48 hours; not improved after 48 hours; use of concomitant medication during trial; total symptoms score; time to recovery; time to return to work; patient assessment of success; temperature and presence of aches, headache, shivers, back or side pain, joint pain, spinal pain, cough, rhinitis, sore throat on evening of day two; fever calculated from temperature using normal distribution   Notes Method of calculating proportions experiencing symptoms described in the text   Allocation concealment A – Adequate

Advertisement