A canna’ change the laws of physics

Scotty, The Naked Time, stardate 1704.3, Episode 7

Dilutions of Grandeur

Posted by apgaylard on March 8, 2009

bigstockphoto_homeopathic_medicine_788631I’ve just come across an article in a new peer-reviewed open-access journal which claims that, “The use of homeopathy may be a useful adjunct the control of epidemics” based on a public health campaign against Dengue which included homeopathy. 

So I’ve given it a read through, and I must say it’s pretty poor.  Perhaps that’s why it’s published in the International Journal of High Dilution Research (formerly Cultura Homeopática) rather than a proper journal. 

If that assessment seems harsh, I hope that the following will bear it out. 

Anyway, Dengue is a pretty unpleasant disease.  It is an, “acute, infectious, mosquito-borne fever that is temporarily incapacitating but is rarely fatal.”  It may give rise to dengue hemorrhagic fever (DHF) which, if left untreated, leads to a, “usually fatal condition known as dengue shock syndrome”. [Encyclopædia Britannica (2009)

Unfortunately there is no specific therapy, aside from symptomatic relief and the support of blood volume in the case of DHF. [See the CDC Dengue Brochure for more details.] 

The lack of a specific therapy and likelihood that most people will recover anyway are probably among the factors which make this an attractive proposition for an uncontrolled homeopathy trial. 

In this post, I’m just going to focus on the methodology used in the trial and the results that were obtained.  I’m not really interested in the details of the ‘remedies’ or the history of placebo-interventions for the ‘control’ of epidemics. [See here for some comments on why homeopathic interventions of the past are not relevant to the present, let alone the future.] 

The Author and His Aims

The author, Laila Aparecida de Souza Nunes (MD, BC Homeopathy), is the Municipal Coordinator of Collective Health, Municipal Secretary of Health, Macaé, Rio de Janeiro.  According to the paper: 

“From the 1990s on, the Brazilian public health system underwent deep changes in its political, administrative and organizational structure, allowing for innovation in the planning and management of health-care facilities, including administration of non- conventional medicines. In this context, phytotherapy, homeopathy and acupuncture were included within the practices offered to users of the national public health system (Sistema Único de Saúde – SUS) in Macaé county, state of Rio de Janeiro.” [Emphasis mine]

So it’s clear that the author has a particular agenda.  This is not necessarily a problem – as long as it doesn’t cloud his judgement.  However, the presence of any dubious data handling may indicate that he has not been able to look at the subject with sufficient objectivity. 

The stated aims of the study were: 

“[…] to assess the contribution of homeopathy to the control of the outbreak of dengue in Macaé in 2007, as a complementary strategy and significant coadjuvant, seeking to evaluate its effectiveness and to measure the acceptance by the general population.” 

Essentially this is posing the questions: would adding homeopathy to conventional interventions make a difference?  And, would people take the homeopathic pills? 

How The Trial Was Done

This is essentially an uncontrolled, open, clinical trial.  Two “Campaigns” were undertaken, in April and November of 2007. 

Homeopathic remedies were dispensed (with conventional interventions) along with “Educational materials”.  Health workers also received specific training and, “a “Routine for the assistance of patients suspected of dengue” included a protocol for the use of homeopathy.”  This resulted in the dispensing of 156,000 ‘doses’ between April and May 2007 and 60,000 ‘doses’ between November and December 2007.  (Estimated population of Macaé was 180,000 inhabitants). 

How sure anyone can be sure that people took the homeopathic potion is unclear: the author states that, “Adherence by the population was spontaneous or voluntary”; whatever that means. 

In addition remedies were given to 129 symptomatic patients.  In these cases the, “disease was diagnosed at the moment of receiving the homeopathic remedy or later, through clinical evaluation.” They were seen at an outpatient clinic and provided, “a 5 ml. vial of homeopathic remedy, to take 5 drops p.o. 3 times a day for one week.”  Again, how can anyone be sure they complied? 

Given that this study is published in the International Journal of High Dilution Research it’s no surprise that the levels of dilution are paradigmatic: 

“All three remedies were used in homeopathic dilution 30cH [10-60], due to the safety guaranteed by a dilution surpassing Avogadro’s number, making virtually insignificant the possibility of side-effects” 

Of course, by surpassing the Avogadro limit to this degree there is in all liklihood no ‘active’ ingredient left in the remedies. 

So, to summarise, there were no: controls, blinding or anyway of knowing if the patients had taken their remedies.  And the homeopathic intervention was delivered along with standard practices:

“[…] control of the vector, epidemiological surveillance and training of both health workers and the general population […] supportive and pharmacological management, as well as clinical follow up […]”

As the author states, “confounding factors were not controlled for”.  I’ll say. 

Given the methodological limitations on view, the results would need to be very spectacular indeed for any conclusions to be legitimately drawn in favour of homeopathy. 

And Now The Results  

Starting with the symptomatic patients, the author reports:

“[…] Follow up showed they presented relief of symptoms from the very beginning of treatment and complete remission was earlier (average = 5 days), without complications, by comparison to patients who did not use homeopathy (average = 8.6 days). None of the homeopathically treated patients progressed to DHF and only 2 were hospitalized, for intravenous hydration.” 

This is effectively a small open-label clinical trial.  The author is claiming a 3.6 day improvement in remission compared to an unspecified control group.  Who were the patients who didn’t use homeopathy? 

Given that the campaign initially delivered 156 000 ‘prophylactic’ doses to in population of around 180 000, I would really like to know how these people escaped.  Were they given doses and not comply – that may indicate a tendency to ignore health messages.  Were they missed by the campaign – missing out on mosquito-bite avoidance advice perhaps?  Maybe they had received their ‘prophylactic’ doses, but chose not to take treatment? 

Perhaps it wasn’t offered to them – in which case I would like to know how that choice was made: were the ‘clinicians’ blinded?  Was allocation to homeopathy random, or the more tractable cases cherry-picked?  We don’t know because the author hasn’t reported on this important aspect of his work.

Neither do we know how many were in the ‘control’ group or how similar they were to the 126 ‘treated’ cases. 

Also, this data was collected during follow-up.  I wonder how long after the patients experienced remission this was.  The author doesn’t say.  Given that they knew they were getting a ‘medicine’ how can confirmation bias be accounted for? 

Finally, no supporting statistical data is provided.  There is no indication whether the mean remission times are (statistically) significantly different. 

In the presence of such poor reporting, how can this result be trusted? Next the ‘global’ results: 

“The homeopathic intervention was followed by a dramatic fall in the number of cases when compared with previous years. (Figure 1) From January to March 2008, 35 cases of dengue were reported in Macaé. In all of them, the disease was benign and no deaths were reported.”

Comparing against previous years is problematic.  This is a mosquito-borne disease, so weather will have a strong bearing: were all years equal in terms of the mosquito population?  Also, this trial is part of a campaign which seems to have been sparked by a bumper year for Dengue cases in 2007.  Were there similar campaigns in previous years?  From the data provided it is clear that the number of reported cases varies considerably year to year. 

I have plotted the data from Figure 1 (below).  It’s clear that 2007 was far and away the worst year, of those reported.  It’s also clear that there were very few reported cases in 2004 and 2005: only 31 and 17, respectively.  This, apparently, was without any help from homeopathy. 

Monthly report of cases of dengue 2002-2007 in Macaé.  Reconstructed from the data provided under Figure 1 in de Souza Nunes (2008)

Monthly report of cases of dengue 2002-2007 in Macaé. Reconstructed from the data provided under Figure 1 in Nunes (2008)

There also seems to be some temporal cherry-picking going on here.  Why choose to aggregate January to March, when the previous April and May were worse? 

Report of cases of dengue during January to March 2002-2007 in Macaé.  Data taken from Nunes (2008)

Report of cases of dengue during January to March 2002-2007 in Macaé. Data taken from Nunes (2008)

Anyway, comparing the same period for the other years shows that three of the six previous years had fewer reported Dengue cases during January to March than 2008. (2004: 12; 2005: 5 and 2006: 10).  From this data there doesn’t seem to be anything remarkable about 2008.

“[…] Annual variation in cases reported in Macaé mirrored the figures reported in the state of Rio de Janeiro from 2000 to 2007, when there was a higher peak in Macaé – triggering the homeopathy campaign. From that moment on, a dramatic fall was recorded, not observed in the remaining localities (Figure 2). The same phenomenon could be observed in the north of the state (where Macaé is located) (Figure 3) […]”

 Figure 2 is interesting for a number of reasons.  It has Macaé’s full-year data for 2008* – so why wasn’t this used instead of the first-quarter comparison made in the paper?  There again, that’s a bit odd because the paper was submitted on 21st November 2008 and published during December – perhaps these data are not for the full year? Interestingly the data go back to 2000 – why does Figure 1 only go back to 2002? 

It also shows that Macaé tends to have lower numbers of reported cases, on the average, than the other regions.   In fact, fewer cases were reported in 2000 and 2005 than in 2008.  Also the result for 2006 seems pretty close to 2008. 

The optimistic homeopath says that, “From that moment on, a dramatic fall was recorded, not observed in the remaining localities” but he is ignoring similar steep falls year-to-year in other areas: Litoral Sul Fluminense between 2006 and 2007 for instance. 

Figure 2 shows that reported cases vary year-to-year on something like a five-year cycle.  There’s pretty strong evidence that all we have here is an example of regression to the mean. 

Anyway, our optimistic homeopath ploughs on heedless of his own data: 

“[…] To compare rates with other counties, we chose those in the state of Rio de Janeiro that in January 2007 had an incidence of Dengue similar do Macaé’s, i.e. 20 cases/100,000 residents. (Figure 4) In these counties, Dengue rates rose dramatically between January 2007 and January 2008, whereas the rate in Macaé plummeted by 93%. (Figure 5) […]”

Setting aside the issue of random variation, another obvious question to ask here is: were there equivalent non-homeopathic campaigns in these counties?  

Figure 5 is odd.  I read it (and I could be wrong) as recording 29 cases in Macaé for 2008**.  This is not consistent with the claim of 35 cases in the first three months or the annual totals shown in figures 2 and 3, which look like being around a couple of hundred. 

Finally, the 93% drop in Dengue cases claimed between January 2007 and January 2008 is not unprecedented: 2003 to 2004 shows exactly the same percentage decrease. 

I must say that I am not convinced.  The only piece of statistical analysis (working out a couple of means) lacks virtually all the relevant details needed to make sense of it (SDs, CIs, any details about the control group). 

The commentary on the graphed data ignores trends and results that don’t fit with the authors disclosed position on homeopathy. 

There is no evidence here for anything other than the confluence of statistical variation, cherry-picking, confounding and wishful thinking. 

What Does the Author Conclude? 

Unsurprisingly for a homeopath he takes the view that: 

“The use of homeopathy in outbreak control complemented more conventional public health actions carried out by the Health Secretary of the county of Macaé […]” 

It is important to note that there is absolutely nothing in this paper to support the homeopathic triumphalism sometimes associated with discussions about the control of epidemics.  This is about homeopathy as an adjunct, not an alternative.   

Even so, there isn’t enough in the paper to make that limited case.  If all the reporting deficiencies were corrected, it is clear from the evidence provided that what happened to the incidence of Dengue in Macaé between 2007 and 2008 is entirely unremarkable.  Add in the uncontrolled confounding factors and the evidence weakens further. 

Now Nunes adds his final flourish: 

On this occasion, considering that health is an integrated activity, the creation of a Nucleus of Integrative Therapies was approved, providing professional health assistance in homeopathy, acupuncture, phytotherapy, homeopathic and phytotherapic pharmacy. 

I think that only a CAM advocate (or someone whose livelihood depended on it) could see extraordinarily weak evidence ‘for’ homeopathy as justifying other, entirely unrelated – and in the case of phytotherapy contradictory – complementary therapies. 

The contents of this paper don’t justify the use of homeopathy – let alone anything else. 

In the interests of fairness I’ve e-mailed the author and posed the following questions: 

  1. Did the other regions, which you compare Macaé against, benefit from a similar public health campaign against Dengue?*** (By similar I mean having all the characteristics of the campaign in Macaé expect for the use of homeopathy)
  2. You say that, “From January to March 2008, 35 cases of dengue were reported in Macaé” and yet Figure 5 appears to show 29 cases for 2008.  Am I misreading Figure 5?**
  3. Your main claim relates to the first three months of 2008.  What were the rates of Denge for the other eight months?
  4. Given that your paper was submitted on 21st November 2008, how is it that Figures 2 and 3 seem to have the incidence data for 2008? Is this data only for part of the year? If so, how many months?*
  5. Why do you claim that the homeopathy element of your campaign was effective when 2004 and 2005 showed only 31 and 17 reported cases, respectively; and between 2003/2004 the reported cases fell by 93%?
  6. When discussing the symptomatic patients treated with homeopathy you say that: “remission was earlier (average = 5 days), without complications, by comparison to patients who did not use homeopathy (average = 8.6 days)”.  What were the standard deviations for these averages?  What were the (95%) confidence limits?  Were the differences between these two averages statistically significant?
  7. The paper gives no details for the control group used to assess the symptomatic patients. How large was it? How similar was it to the treatment group? How was this group selected? Was allocation to the homeopathic treatment made randomly? What allocation method was used? How long after complete remission were patients in the treatment and control groups visited, to record their count of the number of days this took?

It will be interesting to see what he has to say.

Overall this has been a disappointing introduction to a new peer-reviewed open-access journal.  I like the idea of evidence being in the open – it facilitates debate.  It’s a shame that the quality of the evidence isn’t better. 

Still as the International Journal of High Dilution Research is just a re-branded homeopathy journal the ‘peers’ are just as likely to be homeopathy fans as this author obviously is. 

References

Nunes LAS. Contribution of homeopathy to the control of an outbreak of dengue in Macaé, Rio de Janeiro. Int J High Dilution Res [online]. 2008 [cited 2009 Mar 08]; 7(25): 186-192. Available from: http://www.feg.unesp.br/~ojs/index.php/ijhdr/article/view/315/374.

Another paper on homeopathic ‘control’ of Dengue appears in the same issue:

Marino R. Homeopathy and Collective Health: The Case of Dengue Epidemics. Int J High Dilution Res [online]. 2008 [cited 2009 Feb 19]; 7(25): 179-185. Available from: http://www.feg.unesp.br/~ojs/index.php/ijhdr/article/view/312/373.

An excellent critique can be found on Michael Grayer’s nontoxic blog.

Some replies by the author and the content of a Letter submitted to the journal are discussed in the post: Dengue: magic water and the great social mobilization

Acknowledgements

al_capone_junior on the Bad Science forum for pointing out the three month cherry-picking. 

Mojo on the Bad Science forum for the reference to Marino (2008) and Michael Grayer’s critique

Edits

8th March 2009. Two more questions added to the list submitted to the author; querying the use of apparantly full-year data for 2008 when the paper was submitted before the year-end and asking for detials of the control group used for the symptomatic patient comparison.

 

9th March 2009. Reference to Marino (2008) and Michael Grayer’s critique added

13th March 2009. Some random spelling mistakes corrected.

2nd May 2009. Author’s clarifications added as notes.

Notes

The author has kindly clarified a number of issues raised in this post. These notes provide his unedited explanations.

*Figures 2 and 3, the chart shows the updated data until April 16, 2008.

**35 cases was the absolute number, until March 30 and 29 was the incidence rate is calculated per 100,000 inhabitants, and was last updated on 04 April 2008.

***The regions and municipalities perform the vector control, epidemiological surveillance and health education work in a manner similar to Macaé. The difference is the use of Homeopathy and the great social mobilization

In addition to the distribution of homeopathic medicine, intensive work was done for education in health for the population in the control of the vector, the symptoms of the disease, the flow of care, the expected effect of homeopathy as prophylaxis, and dissemination of dial-dengue, which is a telephone in which the user complains that may have local outbreak of mosquito, and is also for people with dengue notify us to take the measure of the vector control and also the “lock” with the homeopathic medicine, as a blocking “vaccine” in the patient’s residence and the neighborhood. So the shares are made, many times before that health services make the reporting of the disease. It also made the mapping of the areas with the highest rate of infestation of the larvae and greater number of cases of disease in these places and make joint efforts to identify and eliminate the outbreaks, guidance on the control of mosquitoes and also with the prophylactic use of homeopathic medicine. With all this, many people become aware of prevention methods in relation to dengue and seek health services as soon as symptoms begin, thus avoiding agravações disease.

[BPSDB]

17 Responses to “Dilutions of Grandeur”

  1. alcaponejunior said

    Hey thanks for mentioning me! Great blog!!

    I’m no expert on medicine, but the cherry picking stood out right away. The whole article was obviously written by someone who knows how to make something look and sound scientific, but the science seemed to be very lacking from sentence one all the way to the end.

    al

  2. apgaylard said

    al: No problem; I’d missed that on my first read-through. Credit where it’s due.

    The more I read the paper the less I liked it. Seems to me that it is about justifying his actions rather than seeing what the data were saying.

    Thanks for your comment.

  3. Does the ‘High Dilution’ refer to the quality of the research?

    After reading your excellent post I’ve just been skimming through the paper and it is eye-wateringly poor. It’s utterly worthless as research but it does highlight again homeopathy’s rather paradoxical craving for scientific credibility.

  4. apgaylard said

    northerndoctor: I can’t escape drawing the same conclusions. I find this whole alt.med journal concept quite intriguing. They look like real journals; they have all the processes and form of the real thing – just not the content. All so very cargo-cult.

    Nice example that the process of peer review really does turn on the quality of the peers!

    Thanks for your kind comment.

  5. […] orker ikke gå i detalj, men det har apgaylard gjort, og han kan fortelle blant annet at: So, to summarise, there were no: controls, blinding or anyway […]

  6. pleick said

    It’s the usual pattern found in so many articles in journals on homeopathy or other CAM methods, isn’t it? The conclusions reached are, at best, an extraordinarily naive and optimistic interpretation of the actual data. The paper shows, once again, that it isn’t enough to just read and cite the abstracts.

    Was there a specific reason that made you look into this paper? I just had a very quick and summary look at it, but judging from this and from your comments, the study doesn’t seem like one that will have any kind of lasting impact.

    I did find a really interesting soundbite in the editorial of the first issue. The editor discusses the reasons for publishing the journal and for making it open-source. The paradox that northerndoctor hinted at – the simultaneous craving for scientific credibility and the ambiguity about the scientific method – is openly adressed. One of the main reasons for homeopathic research and its publication, apparently, are “[…] potential patients, becoming more critical, requiring from doctors a position no longer dogmatic. In other, more cynical words: marketing. To be fair, it’s not the only one.

  7. apgaylard said

    pleick: I picked up the reference from Homeopathy4health and blogged it because, to be honest, it was pretty evident when I looked through it that it wouldn’t be very hard to pick holes in it. I was also a bit interested as the journal was new to me and it’s nice to be able to see the full text.

    I agree that it shouldn’t have a lasting impact – what pro-homeopathy study should? Still, odds are that someone is going to quote it in a debate at some time …

    Your last point is very interesting. Perhaps the only way to limit the damage is to get critical (in the best sense) letters published in the same journals. At least any googling will turn up some balancing information. In this journal it also seems to be possible to submit a comment on-line. I’ll be using both of those routes – after giving the author a decent chance to get back to me.

    Nice to hear from you again.

    BTW. Mojo on the Bad Science forum has pointed out a companion paper in the same issue, which has been debunked by this excellent blog post.

  8. michaelgrayer said

    Hi all,

    Thanks for the nice words, and the linkages to my blog. I didn’t realise that people had actually been reading it!

    A major worry arising from research like this is that it’s the sort of paper that suckers desperate governments into legitimising quack remedies and parting with their taxpayers’ cash. The Brazilian government has (allegedly) given the all-clear for “Proden”, a homeopathic dengue remedy to be made available over-the-counter at pharmacies.

    In an odd way, I have to admit it takes some skill to write papers as bad as these. They’ve made ballsing-up the statistics into an art-form. I challenge anyone here to produce a paper with such a high degree of absolute nonsense as this. I certainly couldn’t do it!

  9. apgaylard said

    michael Thanks for spotting the approval of a homeopathic ‘remedy’ for Dengue in Brazil and blogging it. Your two pieces put my contribution in an more worrying context.

    It seems the use of placebo therapies for serious conditions in vulnerable populations is a worrying trend. The work of various homeopathic organisations in Africa is another example. It’s easy to see the temptation of a cheap alternative to proper interventions; but ultimately if it doesn’t work it’s a dead-end. Better if the emphasis was on fibdubg ways to afford effective therapies.

    Thanks for your comment.

  10. michaelgrayer said

    “It’s easy to see the temptation of a cheap alternative to proper interventions; but ultimately if it doesn’t work it’s a dead-end.”

    I think that’s a very true statement, and it highlights the need for greater awareness in the West of the extent of the problems dengue causes. The major thing that the battle against malaria and HIV/AIDS have going for them is a high profile and good sources of funding for research (the Gates foundation for example). Dengue receives far less attention from legitimate research and thus there is left a wide open goal for snake-oil merchants to use as an easy market opportunity.

    I’m glad that there are more bloggers raising this particular issue.

  11. thesciencepolice said

    And for one of the most notorious dilution experiment exposes of recent history, check out Maddox, J., Randi, J., & Stewart, W.W. (1988). “High-dilution” experiments a delusion. Nature 334, 287 – 290.

    Which we’ve also found cause to reference recently here:
    http://thesciencepolice.blogspot.com

    Working with Citizens to keep the streets safe,
    The Science Police

  12. apgaylard said

    thesciencepolice: Couldn’t find the reference to the Benveniste de-bunking, have I missed something? You might want to check out my “memory of water” page for more dilution delusions. One of my favorite quotes is in a letter to the cargo-cult journal Homeopathy by someone involved with the work called Francis Beauvais:

    “[…]The main issue was that in some circumstances, “effect” and “no effect” were randomly distributed regardless their origin (negative or positive samples) […] the results of blinded samples were almost always at random and did not fit the expected results: some “controls” were active and some “active” samples were without effect on the biological system […]“

    As Swift (I think) put it, “There are none so blind as those who will not see.”

    Thanks for the comment; cool nom de guerre by the way.

  13. thesciencepolice said

    Apgaylard: We had cause to reference Maddox, Randi & Stewart 1988 because it included a quote from J. Randi (a.k.a., The Amazing Randi) that is a guiding principle of The Science Police approach to law enforcement:

    “Look, if I told you that I keep a goat in the backyard of my house in Florida, and if you happened to have a man nearby, you might ask him to look over my garden fence, when he’d say, ‘That man keeps a goat’. But what would you do if I said, ‘I keep a unicorn in my backyard’?”

    We’ll send a copy of the manuscript for your reference. It is a report of the authors’ (thankless) attempt to document how Beneviste’s lab worked and how he, his lab technicians, and the homeopathy community, could come to believe that their published statements were justified. It is full of priceless quotables, very much in the same spirit as your Beauvais excerpt.

    Based on your recent blog posting, and others like it, we unfortunately conclude that these specific instances are not unrelated flukes — and that what Maddox et al reported 20 years ago may still hold true in many alleys in the science world today.

    Back to work,
    The Science Police

  14. draust said

    I quoted the same remark of Randi’s that Thesciencepolice refer to as sort of the punchline of one of my earlier posts which had a similar theme:

    Journals of Alternative Medicine: Insufficient Scepticism = Cargo Cult Science

    Or: Plus ça change, plus c’est la meme chose

    – starring the “Mellow rats in pyramids”, with walk-ons for the late Jacques Benveniste (boo), Richard Feynman (science guru Nobel prize winner) and James Randi (magic man and debunker)

  15. apgaylard said

    draust: Now you come to mention it I do recall that excellent post. Having seen that International Journal of High Dilution Research is a cargo-cult pseudo-journal, I intend to see if they will at least (like Homeopathy) publish something critical.

  16. draust said

    Best of luck.

    BTW, did you see that the Brazilian paper got a plug on the blog of our “friend” homeopathy4health? Or maybe this was where you spotted it.
    I was struck, (though not surprised) by the total lack of any sort of analysis.

  17. apgaylard said

    draust: You are right – that’s where I picked up the reference. The post over there is noting more than “It’s a paper that supports homeopathy – so why even scratch the surface.”

Sorry, the comment form is closed at this time.

 
%d bloggers like this: