A canna’ change the laws of physics

Scotty, The Naked Time, stardate 1704.3, Episode 7

Quacking About Ducks

Posted by apgaylard on November 23, 2007

In the furore that has followed Ben Goldacre‘s trenchant deconstruction of homeopathy several homeopathic apologists have stepped forward.

Denis MacEoin, a novelist and a former lecturer in Islamic studies, writing in the Guardian’s “Comment is Free section ably demonstrated that you get what you pay for.  

It is interesting, in passing, to reflect that MacEoin styled himself a “sceptic” with sympathies for a “pro-science” stance when, according to his wiki entry he has been “…An advocate of alternative medicine since the 1960s… For many years, until its demise in 2003, he was chairman, then president of the Natural Medicines Society …”

Anyway, what really caught my eye was a comment on this piece by the well known US homeopath Dana Ullman.  He, among other things, asked an interesting question:

” … Why do skeptics ignore the fact that three large double-blind placebo controlled trials were conducted on patients with influenza, all showing that a homeopathic medicine called Oscillococcinum was effective. Even the highly respected Cochrane Report acknowledged that these results were “promising.” …”

It is gratifying to note that Ullman refers to the “… highly respected Cochrane Report …”.  It is good to see a homeopath acknowledge the worth of this evidence based medicine resource.  It seems only fair to test his comment against what that “… highly respected …” document contains.

First a simple point of clarification, at least some sceptics are quite happy to engage with this evidence (see here and here, for example).  In fact, this ‘evidence’ has already been debated with Ullman at some length.  So it’s just not true that those of a sceptical disposition “ignore” this information.  It would seem we engage with it frequently.

This is, perhaps, because this ‘evidence’ just keeps cropping up.  It’s also a favourite of Peter Fisher, the Queen’s homeopath.  In an article, published as part of this year’s Homeopathy National Knowledge Week, entitled: “Is homeopathy useful in infectious conditions?” he cited it in some detail (emphasis mine):

“… Influenza has been the subject of the largest number of RCTs of homeopathy of any infectious condition and a Cochrane Review of homoeopathic Oscillococcinum and similar homeopathic preparations for preventing and treating influenza and influenza-like syndromes has been published. Oscillococcinum is made from wild duck heart and liver, and wild fowl are vectors of influenza viruses. The review  found three prevention trials (n= 2265) and four treatment trials (n = 1194). There was no evidence that homoeopathic treatment can prevent influenza-like syndrome, but Oscillococcinum treatment reduced the length of influenza illness by 0.28 days and increased the chances that a patient considered treatment to be effective (RR 1.08; 95% CI 1.17 to 1.00). The authors concluded that the data were promising, but not strong enough to make a general recommendation to use Oscillococcinum for influenza and influenza-like syndromes. Further research is warranted. “

We can also contrast Ullman’s use of the evidence with this statement.

The three prevention trails actually contain nearly twice the number of patients as the four treatment trials.  The result: no evidence that homeopathy can prevent influenza-like syndrome.

Here we can address part of Ullman’s question.  There were not “three large double-blind placebo controlled trials …” , there were four relatively small ones (nave =  298.5).  He must be getting confused with the three larger prophylaxis trials (nave =  755).

The four treatment trials are listed in Table 1.

Study Participants (n)
Casanova 1984 100
Casanova 1992 300
Ferley 1989 487
Papp 1998 372

Table 1.  The Treatment Trials and Their Sizes

In fact, the Cochrane Review authors estimated that confirmatory trials would require sample sizes of 2,000 to 50,000 to detect what they called the “very moderate benefit” of Oscillococcinum; depending on the outcome of interest.

This puts the description of these trials as “large” into some perspective.

Were these trials “… conducted on patients with influenza …” as Ullman would have it?  No; Fisher calls the ailments “…influenza and influenza-like syndrome …” with good reason.

The Cochrane Review notes that the treatment studies recruited “Patients … presenting with influenza or influenza-like syndromes (symptoms of influenza, such as cough, fever, chills and muscle pain, in the absence of laboratory evidence of infection).”

It also made the following comments

  • “… In two of the four treatment trials, patients had to meet a defined standard for influenza-like syndrome (for example, rectal temperature more than 38 oCelsius and at least two episodes of headache, stiffness, lumbar and articular pain or shivers) …” So in two they did not.

  • “… Only one of the treatment studies (Ferley 1989) explicitly reported that patients were accrued during an outbreak of influenza …”

So, it is not clear how many of the patients actually had influenza.  It may appear to be nit-picking, but given the vehemence of Ullman’s contribution he cannot really expect any quarter.

Anyway, back to Fisher’s more temperate piece.  What are we to make of the comment that “…Oscillococcinum treatment reduced the length of influenza illness by 0.28 days …”? The Cochrane Review includes data on statistical significance: “…Oscillococcinum treatment reduced the length of influenza illness by 0.28 days (95% CI 0.50 to 0.06).”

Well, a reduction of 0.28 days in the duration of the illness seems very modest.  This average of 6 hours 43 minutes is statistically significant (at the 5% level), but is this really practically significant?  It’s worth noting that the Cochrane Review comments: “This effect might be as large as half a day and as small as about an hour.”

What about increasing “… the chances that a patient considered treatment to be effective (RR 1.08; 95% CI 1.17 to 1.00) …”?  This is a very small treatment effect, based on the patient’s subjective impression of their symptoms.   Looking at the Confidence Intervals we can actually say that this result is not statistically significant at the 5% level.  Why?

Here’s a very clear explanation from the Link Mobile Telecommunications and Health Research Programme (MTHR) Report (2007), (emphasis my own):

“… The effect that chance can play on the odds ration obtained in a study is usually expressed by the 95% confidence interval (CI), which is the range of odds ratios consistent with the data allowing for statistical variations.  The 95% CI indicates that, if 100 similar studies were carried out, the odds ratio found in 95 would be expected to lie within the CI, while in 5 they would lie outside by chance.  In general, if the CI includes 1.0, the results are said not to be significantly different from 1.0, ie not statistically significant: no convincing evidence of a statistical association …”

The CI quoted by Fisher includes 1.0; we have a statistically insignificant result at the 5% level.

It’s also worth considering this comment from Ioannidis:

“… The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. Power is also related to the effect size. Thus research findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3-20), than in scientific fields where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1-1.5) …  In the same line of thinking, if the true effect sizes are very small in a scientific field, this field is likely to be plagued by almost ubiquitous false positive claims. For example, if the majority of true genetic or nutritional determinants of complex diseases confer relative risks less than 1.05, genetic or nutritional epidemiology would be largely utopian endeavors …”

This review is dealing with quite small effect sizes and is thus likely to be suffering from the tendency to false positive claims Ioannidis describes.

Both Ullman and Fisher referred to the Cochrane Review.  What did that conclude?  Its “plain language summary” states: “Homoeopathic Oscillococcinum does not prevent influenza but might shorten the length of the illness … Trials do not show that homoeopathic Oscillococcinum can prevent influenza. However, taking homoeopathic Oscillococcinum once you have influenza might shorten the illness, but more research is needed.

So, as we have noted: no prophylaxis, but maybe a small shortening of the illness’ duration.

Ullman contends that the medication was found to be effective.  It was not.  The patient’s slightly positive view of the preparation’s effectiveness is not statistically significant.  If it was effective the authors would have recommended its use.

Fisher’s more honest appraisal of the review notes that “… The authors concluded that the data were promising, but not strong enough to make a general recommendation to use Oscillococcinum for influenza and influenza-like syndromes. Further research is warranted.”

There are other important details in the Cochrane Review.  These, no doubt, underlie the author’s cautious conclusions.

“… The standard of trial reporting was poor. For only two studies was there sufficient information to complete data extraction fully (Ferley 1989; Papp 1998) …”

One of the treatment trials (Casanova 1992) was unpublished; the other (Casanova 1984) “… was reported in a general medical magazine rather than in a scientific journal.”

The Review also found the sample sizes in the two Casanova papers to be “… suspiciously round numbers (100 and 300) …”

So, in summary, we can answer Ullman’s question as follows.

  1. We don’t ignore the data.  Many of us are happy to engage with it.  He knows this.

  2. Four relatively small studies were conducted.  Two were by the same author.  One of these was unpublished, the other published in a general medical magazine.  Both had, what the Cochrane Review called “…suspiciously round numbers …” of participants.

  3. It’s not clear that all the patients had influenza.  In the treatment trials, these were “Patients … presenting with influenza or influenza-like syndromes …”

  4. The homeopathic medicine was not effective.  The Review states that “… taking homoeopathic Oscillococcinum once you have influenza might shorten the illness, but more research is needed.”  The slight positive outcome for the patient’s view of effectiveness was not statistically significant.

  5. The results were called “promising“, but under “implications for practice” the Review stated that “… the evidence was not strong enough to make a general recommendation to use Oscillococcinum for routine treatment …”

Fisher’s use of the data is clearly more considered and honest.  However, it is disappointing that he quotes a statistically insignificant result without comment.

Summary Tables For The Treatment Trials Extracted From The Cochrane Review.


 

Study

Casanova 1984


 

Methods

Treatment allocation: B
Performance bias: B
Observer blinding: B
Exclusions/withdrawals: B


 

Participants

100 patients with influenza-like syndrome onset less then 48 hours previously. No details of method of recruitment or exclusion criteria. Average age verum/placebo: 42/41 years. Male:female in verum/placebo: 19:31/26:24


 

Interventions

Oscillococcinum, four doses in over two days at six-hour intervals


 

Outcomes

Patient global assessment of success; presence of chills, aches rhinitis, night cough, day cough, fever at day eight


 

Notes

Reported in what appears to be a general medical magazine: very few experimental details given


 

Allocation concealment

B – Unclear

 


 

 


 

Study

Casanova 1992


 

Methods

Treatment allocation: B
Performance bias: B
Observer blinding: B
Exclusions/withdrawals: B


 

Participants

300 patients complaining of influenza. No details of inclusion or exclusion criteria. Average age verum/placebo: 44/38. Male:female in verum/placebo: 61:89/56:94


 

Interventions

Oscillococcinum, twice a day for three to four days


 

Outcomes

Temperature recorded twice a day for four days (data for evening of second day used for continuous outcome, data for evening of day four converted to binomial outcome of fever, by using normal distribution); presence of chills, aches at day four


 

Notes

Inconsistency between text and table: the table appears to have been printed the wrong way around. The text value was selected


 

Allocation concealment

B – Unclear

 


 

 


 

Study

Ferley 1989


 

Methods

Treatment allocation: A
Performance bias: A
Observer blinding: A
Exclusions/withdrawals: A


 

Participants

487 patients presenting in primary care with a complaint of influenza-like syndrome. Inclusion criteria: age older than 12 years; rectal temperature above 38 Celsius and at least two of headache, stiffness, lumbar and articular pain, shivers. Exclusion criteria: duration more than 24 hours; immune deficiency; local infection; immunisation against influenza; depression; immunostimulant treatment. Average age verum/placebo: 34/35. Male:female in verum/placebo: 93:127/97:129


 

Interventions

Oscillococcinum, twice a day for five doses


 

Outcomes

Patient assessment of success; recovery at 48 hours (defined as rectal temperature below 37.5 Celsius and complete resolution of all five symptoms); number of days to recovery; number of days to return to work; use of medication for pain or fever; use of medication for cough or sore throat; use of antibiotic medication; patient assessment of success


 

Notes

Use of medication calculated from percentages given in text. Some minor inconsistencies between figures suggest a small amount of missing data


 

Allocation concealment

A – Adequate

 


 


 

Study

Papp 1998


 

Methods

Treatment allocation: A
Performance bias: A
Observer blinding: A
Exclusions/withdrawals: A


 

Participants

372 patients recruited in primary care or by internal medicine specialists. Inclusion criteria: rectal temperature above 38 Celsius; muscle pain or headache; one of shivering, cough, spinal pain, nasal irritation, malaise, thoracic pain, periarticular pain. Exclusion criteria: duration more than 24 hours; immune deficiency; local infection; immunisation against influenza; medical need for medication; immunostimulant or immunosuppressive treatment.
Use of analgesics, antibiotics or anti-influenza agents in the first 48 hours was a post-randomisation exclusion criterion. Average age verum/placebo: 35/35. Male:female in verum/placebo: 95:93/96:88


 

Interventions

Oscillococcinum three times a day for three days


 

Outcomes

Complete recovery after 48 hours; not improved after 48 hours; use of concomitant medication during trial; total symptoms score; time to recovery; time to return to work; patient assessment of success; temperature and presence of aches, headache, shivers, back or side pain, joint pain, spinal pain, cough, rhinitis, sore throat on evening of day two; fever calculated from temperature using normal distribution


 

Notes

Method of calculating proportions experiencing symptoms described in the text


 

Allocation concealment

A – Adequate

 


 

About these ads

7 Responses to “Quacking About Ducks”

  1. jdc325 said

    You’ve picked out some real gems there and explained them so well that I think even I managed to follow the points you made. (I found the MTHR explanation of CIs and statistical significance particularly helpful).

  2. apgaylard said

    jdc325: Thanks. I’ve been waiting for a chance to use that passage from the MTHR report.

    It’s also important to note that the idea that this preparation is based on, the liver being the source of fever is nonsense. (Ignoring the like-cures-like and ultra-molecular dilutions)

    The whole thing is an argument about barely (statistically) significant results from questionable studies testing the massively implausible. It’s a field that’s just ripe for false positive results.

  3. Post-holiday “The stupid, it burns,” part 2: Denis MacEoin

    I’ve never been able to understand advocates of homeopathy. I just have difficulty understanding how otherwise intelligent people can fall for the bad science, the logical fallacies, and the magical thinking necessary to believe that homeopathy is any…

  4. hcn57 said

    Very good and detailed takedown of Ullman’s reliance on those “promising” studies. The one problem is that those studies and others have been discussed with him. He posts as “JamesGully” in this thread:
    http://forums.randi.org/showthread.php?t=82393&&page=2

    If you slog through all 27 pages of that thread you will see Ullman slip, slide and eventually run away.

  5. apgaylard said

    Hcn57:

    Thanks for the comment and reference. The point in the thread about multiple comparisions and significance is well made.

    Ullman does seem to go in for drive-by nonsense. Here’s a recent example, I think you commented on the thread?

  6. [...] also links to this fantastic exploration of some of the defenders who have appeared in the Guardian thread, by physicist blogger [...]

  7. hcn57 said

    Thanks, and yes I did comment on that thread. I confess I am a lowly commenter compared to the skill of others (and lazy, I often just go after easy targets like Ullman). I have learned much from Rolfe, BadlyShavedMonkey, Dr. Aust, Orac and others. Sometimes I just enjoy watching them work over a quack.

Sorry, the comment form is closed at this time.

 
Follow

Get every new post delivered to your Inbox.

Join 597 other followers

%d bloggers like this: