Spying on Shang
Posted by apgaylard on August 10, 2008
It seems like I’m becoming a collector of misconceptions about the Lancet paper published by Shang et al (2005). This week I’ve been having a small disagreement with a homeopath named Clive Stuart on Margaret McCartney’s blog at Ft.com.
One of his criticisms of Shang et al was new to me. He said, “When the 8 studies were finally revealed, it turned out that most of them were for the prophylaxis of flu.”
Now, having read the paper in question; the ‘webappendices’; the author’s letter of reply in the Lancet and the additional material they have made available on-line: I was surprised.
I pointed out that there were only two studies in the final eight that were concerned with ‘flu-like symptoms: Rottey (80) and Papp (71) [note: the references are to the webappendix and additional material]. I thought that this would be the end of it, but no, he countered, “You say that “only two related to flu-like symptoms”. This is incorrect. Actually 5 of the 8 studies related to influenza. Three dealt with prophylaxis of influenza and two with actual treatment of influenza. One study dealt with prophylaxis of conjunctivitis, one with treating acute respiratory inflammations and one with muscle soreness in runners.”
Again, the same error; but made with such confidence and additional detail. I wondered whether he might have been misled by an apparently authoritative source. A little ‘googling’ reveals a commentary on “Egger/Shang’s meta-analysis” published by The European Committee for Homeopathy (ECH) in September 2005.
“Our Swiss colleagues who know people who were involved in Matthias Egger’s meta-analysis (Shang et al), have disclosed that the following 8 trials were used for the conclusion of the meta-analysis …
… Three studies deal with the prevention of flu, two with the treatment of flu, one with the prevention of conjunctivitis, one with the treatment of acute respiratory inflammations with a complex remedy and one with the treatment of muscle soreness in long-distance runners.”
Spies in the camp! Very James Bond. Now, the content is so similar that if this is not Stuart’s source, both must share a common source. This document is also claiming to disclose original information, increasing the likelihood of it being Stuart’s primary source.
The date of the document also falls in to the period between late August and December 2005; the interregnum between the publication of Shang et al and the author’s disclosure of the identity of the final eight studies of homeopathy (and the six of conventional medicine).
During this time it’s clear that the homeopathic community was, rightly, concerned to discover the identity of the studies. It seems that their enthusiasm spilled over into desperation and they relied on hearsay. It’s also typically disappointing that a homeopath has failed to keep up-to-date with a topic which seems to be so important to them (it was Stuart who first raised the topic of Shang et al in the thread.)
Are there any general lessons here? First, it illustrates the importance of going back to primary sources, where possible. A corollary of this is that hear-say is not to be trusted.
If Stuart had actually read the Author’s reply he would have found the eight trials that he is interested in identified by a reference number, “larger trials of higher methodological quality (references 46, 55, 71, 80, 84, 94, 96, 97 in webappendix 1)”
Looking at either webappendix 1, or the extra material provided on-line, enables anyone to match the numbers to the trials. And you get this list:
46. Jacobs J, Jiménez LM, Malthouse S, et al. Homeopathic treatment of acute childhood diarrhea: results from a clinical trial in Nepal. J Altern Complement Med 2000; 6: 131-39.
55. Labrecque M, Audet D, Latulippe LG, Drouin J. Homeopathic treatment of plantar warts. Can Med Assoc J 1992; 146:1749-53.
71. Papp R, Schuback G, Beck E, et al. OSCILLOCOCCINUM® in patients with influenza-like syndromes: a placebo-controlled double-blind evaluation. Br Homeopath J 1998; 87: 69-76.
80. Rottey EED, Verleye GB, Liagre RLP. Het effect van een homeopathische bereiding van micro-organismen bij de preventie van riepsymptomen: een gerandomiseerd dubbel-blind onderzoek in de huisartspraktijk. Tijdschr Int Geneeskunde 1995; 11: 54-58.
84. Schmidt JM, Ostermayr B. Does a homeopathic ultramolecular dilution of Thyroidinum 30cH affect the rate of body weight reduction in fasting patients? A randomised placebo-controlled double-blind clinical trial. Homeopathy 2002; 91: 197-206.
94. Vickers AJ, Fisher P, Smith C, Wyllie SE, Rees R. Homeopathic Arnica 30x is ineffective for muscle soreness after long-distance running: a randomized, double-blind, placebo-controlled trial. Clin J Pain 1998; 14: 227-31.
96. Walach H, Haeusler W, Lowes T, Mussbach D, Schamell U, Springer W et al. Classical homeopathic treatment of chronic headaches. Cephalalgia 1997; 17: 119-26.
97. Weiser M, Clasen BPE. Randomisierte plazebokontrollierte Doppelblindstudie zur Untersuchung der klinischen Wirksamkeit der homöopathischen Euphorbium compositum- asentropfen S bei chronischer Sinusitis. Forsch Komplementärmed 1994; 1: 251-59.
It is clear that the conditions covered by these trials are: (46) acute childhood diarrhea (sic), (55) plantar warts, (71) influenza-like syndromes, (80) influenza like disease, (84) body weight reduction in fasting patients, (94) muscle soreness after long-distance running, (96) chronic headaches and (97) sinusitis. For anyone keeping count that’s two trials involving influenza like disease/syndromes (note, the articles don’t claim to be about ‘flu per se as it was not confirmed in the patients.)
Another general lesson that could be drawn is the importance of keeping up-to-date. It seems that Stuart was aware that the author’s had identified the trials, “many months later” (four, actually). However, this had not prompted him to check whether the ‘friends of friends’ of the ECH were correct.
(I should also point out that the ECH appear to be aware of the correct composition of this group of studies, judging from their discussion of Shang et al. However, they don’t seem to have explicitly corrected their earlier ‘intelligence’.)
Stuart’s statements also highlight some of the schisms between the various homeopathic communities, “When the 8 studies were finally revealed, it turned out that most of them were for the prophylaxis of flu !?! This has absolutely nothing to do with standard individualized homeopathic practice.”
Contrast this with the position of other homeopaths on Oscillococcinum, a homeopathic lack of duck heart/liver sold for both the prevention and treatment of ‘flu. According to the Complementary and Alternative Medicine Website of the New Zealand Guidelines Group, “”Homoeopathic preparations, such as oscillococcinum or similar products, are often prescribed by homoeopaths for the prevention and treatment of the flu.” (google “oscillococcinum +prevention +flu” for more examples.)
When variants of homeopathy appear to come out well in trials they are embraced; when there is an apparent problem they are an impure aberration and condemned:
“La vittoria trova cento padri, e nessuno vuole riconoscere l’insuccesso.” [Victory has a hundred fathers, and no one acknowledges a failure.]
The Oxford Dictionary of Proverbs. Oxford University Press, 1982, 1992, 1998, 2003, 2004. Answers.com 10 Aug. 2008.
19 Responses to “Spying on Shang”
Sorry, the comment form is closed at this time.
A canna’ change the laws of physics 
dvnutrix said
Yet – that is not what people want to hear. They/We would rather persist in error than consult an original source or revise an impression based on something that they/we read about it.
I wonder if this is an odd corollary of source amnesia – once you are convinced that something is true, if the correction comes from a source to which you are fundamentally opposed then nothing will make you change your mind. Probably, not even being pointed to a correction made by people who you perceive as ‘being your your side’.
jdc325 said
Clive Stuart’s comments are a joy – I love that he called you on those 8 studies and responded to your “let’s at least get the facts straight first” with “Quite”. I wonder if he’ll acknowledge that he was wrong. It must be a bit embarrassing for him to incorrectly assert that someone is wrong, particularly when it is in so public an arena. Personally, I have always found a swift and polite retraction to be the best course of action when I have made similar errors.
PS – Having read the FT exchange, I can say I was slightly depressed by some of the comments: “Clive, thanks for staying cool and reasoned in this debate. I have seen a number of blogs discussing this issue and have been astonished at the vitriol of the people who are anti-homeopathy” being one. I’m not sure which blogs this commentor has been reading, but it certainly isn’t the same ones I read! Unless there is an alternative definition of ‘vitriol’ that I am not currently aware of…
I also particularly liked the comment immediately below your last – “NHS homeopathy as provided by regulated health professionals, is supported by a good evidence-base and has consistently excellent outcomes”. Where do they get this stuff from? Is there an alternative meaning of ‘evidence base’? (Not to mention alternative meanings of ‘consistently’ and ‘excellent’ – maybe there is a whole alternative dictionary? The Humpty Dumpty English Dictionary perhaps?)
apgaylard said
dvnutrix: Thanks for the reference. I’m not sure whether there is any interesting psychology going on here, or if it’s just poor scholarship. From my brief exposure to CAM world I’d say that the level of discourse is pretty poor. If you get used to those sorts of low standards you’re bound to come a cropper. both could, of course, be true.
jdc325: I couldn’t believe that he was so forthright in his response, given the facts. It’s that which really piqued my interest. My guess (and maybe he’ll explain at some point) is that he really trusted the ECH document and didn’t see the importance of checking for himself. I also loved the whole cloack-and-dagger thing that the ECH had going on.
I’m not so sure that ‘Clive’ has been all that cool in the debate. I’ve seen that sort of ‘sceptics are nasty, we’re being nice’ formulation quite often. I have to say that the True Believers often over-rate their own niceness.
I hope he contributes some more. Like you, I have found that it’s best to retract errors (and I’ve made a few) with some humility.
Thanks both.
dvnutrix said
True. However, I’m constantly shocked at the number of people who effectively say “never mind the quality, feel the width” when it comes to references. They don’t grasp the concept that it is entirely possible to have a thousand or more references, many of which are mis-cited or misinterpreted. When this is pointed out to them, instead of being pleased at the opportunity to correct some mis-beliefs, there is usually hostility.
I rather enjoyed the Gladstone-Manjoo discussion of convenient untruths. It seems that there is a state somewhere between cognitive dissonance and selective interpretation where it is possible to disbelieve anything that doesn’t support you.
Now, in this case, it would be easy for Clive Stuart to check that you are correct and he is not – yet he hasn’t, which is interesting.
As for some people’s interpretation of vitriol (and inaccuracy, come to that)…
wilsontown said
Good digging here.
The whole circus surrounding the Shang paper is utterly bizarre. It doesn’t seem to be a particularly difficult paper to understand, and yet most of the objections to it from homeopaths could be refuted by just reading the damned thing.
gimpy said
I am now firmly of the opinion that homeopathy, of all sorts, is a religious movement akin to young earth creationism. Even the respectable homeopaths like Peter Fisher, the Queens Quack, treat Hahnemann’s (and to a lesser extent Kent’s) writings as inerrant holy writ (despite the fact that the Organon went through 6 versions – homeopaths seem to pick the version that best suits the occasion). Reality conflicts with Hahnemann, then reality is wrong. You can’t argue with a mindset like that. The trouble with homeopaths trying to understand Shang is that they cannot accept the conclusion in any way, shape or form, so the paper must be wrong. This means that rather than critically appraise arguments against the paper they blindly accept them because, y’know, the paper is just wrong.
apgaylard said
wilsontown: It’s certainly another for the collection. Something of an advance on ‘they kept the identities of the studies secret’. For a connoisseur of such things, such as your good self, there are some other potential classics on the ECH website under the ‘research’ menu.
gimpy: Fair conclusion. I think that because the writings of Hahnemann, Kent, Herring are venerated there is an inclination to look at other ‘authorities’ in a similar way.
Thanks both.
apgaylard said
dvnutrix: Interesting definition of vitriol: disagreeing with me or asking for evidence. I take your point about references. I’ve probably mentioned it before, but in the day job I referee papers for several journals/conferences. I’m always careful to check references where I can after stumbling across several incidents of references saying the opposite of what is claimed.
Add in the sloppy citations (interesting New Scientist piece on this some years ago) and you have something like a bad-information virus propagating through the literature.
Sometimes it is possible to get a citation-loop: you try to get back to the origonal source only to end up where you have been.
Thought-provoking as ever. Thanks.
dvnutrix said
Thanks for the New Scientist reference – I’ll dig that one out. I think that one phenomenon is sometimes known as a self-replicating bibliographic virus – and one of the most famous examples is in diabetes research.
We’ve identified some references of Holford’s that don’t exist but he is still using them. And that makes it the more galling/annoying when people repeatedly say, “he has hundreds of references”.
pyjamasinbananas said
That paper referred to in the New Scientist article really winds me up. Who says they didn’t read the paper just because they copied an incorrect reference? Back in the days before we could access references directly from databases we copied our references from wherever came to hand, whether that be the paper itself or someone else’s reference list.
dvnutrix said
Just seen Checking References.
I object to:
references that don’t exist
errors that are so bad I think the details were cribbed from a news-release w/out seeing the paper
gross distortions.
PinB – I haven’t seen the NS item but I have seen people quote results from papers that don’t exist – and it’s not just a mis-citation. This is how the literature is polluted. [/soapbox]
gimpy said
PJ, in the modern internet age there is no excuse for not correctly citing papers. I have to say I am mildly disapproving of my colleagues who confess to not properly checking references when reviewing papers. The integrity of an academic work is dependent on the assumption that an author has correctly cited research in support of their arguments and it is the job of the reviewer to make sure of this.
pleick said
It should be the job of a reviewer to check references. I have to confess, though, that I’ve often taken shortcuts on this and only checked when I had a prior suspicion that something not quite kosher was going on.
The point is that checking references, if done seriously, is extremely time consuming, but not very rewarding. It requires that you actually read the papers and not just the abstracts, because too often, the abstracts just give a rough impression of the papers’ contents. And in the vast majority of cases, you’ll not find anything unusual, since most researchers – at least in the fields I’m familiar with (if you remove quantum esoterics, that is) – are conscientious and honest, and at least try to be accurate with their references.
I completely agree with Gimpy’s comment, but I’ll add that any reviewer who thoroughly checks the text of a paper and takes a quick glance at the references is, in my opinion, already doing a fine job.
As a researcher, I don’t expect my colleagues to try to deceive me… this trust is what makes the scientific enterprise work. For the record, I don’t believe all’s well either (one of my colleagues once admitted to me that he reads about half of the papers he cites – which is probably a rather good figure). But we’re doing pretty well. Mostly.
Of course, certain pseudoscientific enterprises are a completely different matter. Having done this a few times, I can say that checking for “bad references”… well, look, and you shall find.
jdc325 said
Ooh, has anyone linked to Neurologica yet? Via the HolfordWatch miniblog: How Confident Are You?
Sample quote: For the critical thinker it is important to avoid unjustified confidence, and to understand what factors may lead us to a high degree of confidence when we are in fact wrong.
apgaylard said
Fascinating discussion. The paper the NS article comments on can be found on arXiv [pdf]. Their assumption is that if you have read the paper you’ll write your own bibliographic reference, not relying on anyone else’s. Hence, if you get the reference wrong it’ll be your own, original, mistake. I agree with PJiB that this assumption may not always hold up.
However, the authors do address this point:
I’d also add that some errors in the bibliogarphic data can make locating the reference hard to start with. I’ve had librarians (when this luxury existed) query the details that I’d given them when they couldn’t locate the article in a particular journal issue, for example. I’ve had similar problems when I’ve been doing the searching. So I guess that if the error is in the location data then a ‘reader’ ( or, at least, someone who’d sourced the article) would be very aware of it and unlikely not to correct it in their citation.
So, perhaps the study does over-state the problem; but I don’t think that it’s too bad a starting point when trying to get a handle on this problem.
Thanks to all for the well-considered comments.
apgaylard said
Update:
Clive Stewart has returned to the fray with a guarded apology and wild accustations.
I’ve replied twice (here and here) and am mulling some of the issues over for a blog post. More on this later.
jdc325 said
“Inexplicably Shang et al do not rate these studies as high quality even though one of their included studies has less subjects than both.”
He seems to think that the phrase “high quality” relates solely to the number of subjects.
This was cracking, though:
@Stuart: “The information was quoted as being given by individuals who were involved in the study itself. It can neither be disproved or verified.”
@APG: “There is in fact a way to check that the final eight studies were those citied in the Author’s reply – do the analysis and see if the result is the same an in the original paper.” Should make for a good blog post. You aren’t going to ask him to retract every incorrect statement he makes on Shang though are you? I think you’ll find there simply aren’t enough hours in the day…
Cheers,
jdc
apgaylard said
jdc325:
Thanks for your interest. This guy doesn’t get Shang at all. But there again neither do the homeopathic set generally. I seriously doubt whether he has actually read it for himself. If he had he’d know that the quality criteria were not to do with size.
“You aren’t going to ask him to retract every incorrect statement he makes on Shang though are you?” No – as you say, life is too short. I’m not really interested in debating the guy anymore – he just doesn’t get it; don’t think he’s too keen either. He does provide an interesting muse though. I think he illustrates problems a lot of alties have in engaging with the scientific literature. I’ll do that next. I’m debating doing a Shang FAQ; but that could be a big job and I’m really meant to be writing a conference paper at the moment…
Examples of Homeopathic Debate « jdc325's Weblog said
[…] forum and two of the Ullman issues that were listed by AP Gaylard. Gaylard has written that he is becoming a collector of misconceptions about the Lancet paper published by Shang et al. It seems to me that until and unless homeopaths have accurate information about, and a good […]