A canna’ change the laws of physics

Scotty, The Naked Time, stardate 1704.3, Episode 7

The Myth of The Secret Eight

Posted by apgaylard on November 28, 2007

There seems to be a persistent myth among homeopathic apologists that the final eight homeopathic studies (large and of high quality) analysed in the famous Lancet paper by Shang et al. are somehow secret.

Here is a recent example of this piece of homeomythology:

“… When Shang et al (Lancet 2005) did a metaanalysis of 110 homeopathy trials, homeopathy was significantly better than placebo. When they restricted it to “high quality” trials (n=20), homeopathy was still significantly better than placebo, same as the ‘matched’ biomedical trials. They then restricted it to “large, high quality” trials (n=8), and claimed they were no better than placebo. Does anyone here know which 8 homeopathy RCTs were selected? Failing that, the 20 “high quality” trials, from which one could identify the 8 largest. And how and when did they set the cutpoint for “large”?”

“Chamfort”, Comment is Free, Guardian, Comment No. 941849, November 22 13:11

So, is it possible to find which eight trials of homeopathy were finally selected?  Yes.  In fact one poster did provide an answer to this question.  He noted that “…The 8 homeopathy trials and 6 conventional trials that were considered to be large, high quality trials are identified by the authors in their reply to comments on the original article. See Lancet 366 (2005); p. 2083. The list of included trials can be found as an appendix to the online version of the original article, see Lancet 366 (2005); 726-732.”

Where did this idea come from?  Well, Shang et al. did omit these details from their original paper.  This was picked up by many apologists, including correspondents to the Lancet: Fisher et al., Linde  and Jonas, Walach et al., and Dantas.  Shang et al. then provided the information, as noted above, in the same issue of the Lancet as the critical letters.

But, as we have seen, the complete story has been missed by some homeopathic apologists.  It would seem that they only read the critical correspondence, or quotations from it.  This ‘secrecy’ myth has now passed into homeopathic folklore.

Here’s a very recent comment from a pro-homeopathy blog:

“…Another oft-raised issue is that homeopathy cannot stand up to conventional medical trials, and that homeopathic studies fail meta-analysis. It is suggested that homeopaths “cherry pick” the results of studies to confirm positive results. Iris Bell MD Phd, director of research at the Program in Integrative Medicine at the University of Arizona College of Medicine, stated the following in response to a negatively biased article in the British medical journal “The Lancet” in August of 2005;

“The researchers started with 110 homeopathy studies and 110 conventional medicine studies, but drew their main conclusions from 8 homeopathy and 6 conventional studies. And in an odd decision by the journal and the researchers, those 14 studies were not identified. This makes it impossible for others to challenge or confirm their conclusions. Given the tiny number left, they really were under an obligation to tell the reader which they used.” …”

http://goodscience.wordpress.com, 24th November 2007.

This quotes an article that Bell had published in the Journal Of Alternative And Complementary Medicine before the additional information was made available by Shang et al.

So, this criticism gets made over and over again; every time the meta-analysis of Shang et al. is discussed.  It makes me wonder why.  Perhaps perceiving themselves as upholders of an unchanging tradition, homeopathic apologists don’t think that the argument moves on?  Maybe a paranoid world view that includes the great allopathic conspiracy makes them think that such omissions are made with malign intent and evil scientists won’t correct them?  Perhaps it’s just sloppy scholarship?

In fact, quite a lot of detail about the studies included and excluded by Shang et al. is freely available online.  On the website of Shang’s home institution, Institut für Sozial und Präventivmedizin at Universität Bern, there are three files that can be downloaded:

“… List of excluded homoeopathy studies (PDF 32KB) provides a list of excluded homoeopathy studies with reasons for exclusions.

Study characteristics of homoeopathy studies (PDF 72KB) provides details on study characteristics and results of 110 placebo-controlled homoeopathy trials.

Study characteristics of allopathy studies (PDF 72KB) provides details on study characteristics and results of 110 placebo-controlled allopathy trials. …”

This information was originally posted on December 17th, 2005 and subsequently corrected (for an error in the graphs) on December 23rd, 2005.

It’s easy to use the second file to determine the identity of the ‘secret eight’:  select the trials identified as of high quality, list them by size and pick the eight largest.  The results of doing this are shown in Figure 1.  The ‘secret eight’ are obvious; as is the cut-off criterion.

The Secret 8 Revealed

Figure 1  The Secret Eight Revealed

Table 1 provides details of the intervention used, clinical indication and outcome.  Two of the ‘secret eight’ studies were of individualised homeopathy.  Others used single or multiple preparations and two were of proprietary homeopathic ‘medicines’.



High Quality Study

n

Intervention

Indication

Outcome

Rottey (80)

501

Mucococcinum 200K

Influenza like disease

Occurrence of symptoms

Vickers (94)

400

Arnica 30X

Physical activity

Soreness

Papp (71)

334

Oscillococcinum®

Influenza like disease

Global assessment, patient

Schmidt (84)

208

Thyroidinum 30C

Fasting

body weight

Labreque (55)

162

Thuya 30C, Ant 7C, Nitr acid 7C

Warts

Global assessment, physician

Jacobs (46)

116

Individual treatment

Diarrhoea

Duration of diarrhoea

Weiser (97)

104

Euphorbium compositum®

Sinusitis

Global assessment, other

Walach (96)

98

Individual treatment

Headache

Pain

Jacobs (49)

81

Individual treatment, 30C

Diarrhoea

Duration of diarrhoea

Jacobs (47)

75

Individual treatment

Otitis media

Global assessment, other

Hart (41)

73

Arnica C30

Hysterectomy

Pain

Wiesenauer (102)

72

Galphimia glauca D4

Pollinosis

Global assessment, other

Zell,von, (105)

69

Traumeel®

Sprain Joint

angulation

Böhmer (15)

67

Traumeel®

Sports injury

Global assessment, patient

Vickers (95)

57

Arnica 30C, Rhus t 30C, Sarc a 30C

Physical activity

Soreness

Chapman (21)

50

Individual treatment

Traumatic brain injury

Activities

Jawara (50)

50

Arnica 30C, Rhus t 30C

Physical activity

Soreness

Tveiten (91)

46

Arnica D30

Physical activity

Soreness

Stevinson (86)

42

Arnica C6

Hand surgery

Pain

Lepaisant (58)

36

Folliculinum 9C

Premenstrual syndrome

Global assessment, other

Chapman (22)

10

Individual treatment

Premenstrual syndrome

Global assessment, other

Table 1.  Details of the High Quality Homeopathic Studies Identified by Shang et al

The use of single preparations, or set ‘medicines’, in clinical trials is often criticised by homeopaths; however they seem happy enough when the results appear to be  positive.

I have also yet to see evidence of concerted criticism from the homeopathic community of high street retailers like Boots selling ‘off the shelf’ homeopathic preparations for treating particular conditions.

Homeopathic apologists do tend to miss the point of Shang et al.  This analysis demonstrates (again) that removing bias, trials of low quality and small size progressively diminishes the apparent clinical effects of homeopathy to the point where they cannot be distinguished from a placebo.  This is just what would be expected if homeopathy was just a placebo.  Any apparent effects in excess of this are just so much statistical noise.

So, finally, let’s dispel this particular piece of homeomythology:  the identity of the final eight, high quality relatively large, homeopathic trials selected by Shang et al. are not secret.  They have been in the public domain for nearly two years.

About these ads

9 Responses to “The Myth of The Secret Eight”

  1. Nice digging.

  2. wilsontown said

    Excellent stuff. Was going to write something on this myself, but now I think I needn’t bother.

    I’ve yet to see a critique of the Shang et al. meta-analysis that stands up to scrutiny, and I’ve been looking.

  3. apgaylard said

    Wilsontown: Thanks. This one has been on my mind for a while. I must have dowloaded the PDF files over a year ago. Thanks for your CIF post; it galvanised me into action!

  4. Not a moment too soon, because here is Dana Ullman:

    Perhaps, SOMEONE can finally tell us which were the 21 homeopathic trials and the 9 allopathic ones. Shang NEVER divulged, most likely because this review would show real benefits from homeopathic treatment. Isn’t anyone suspicious of “black box” comparison studies like this? Why are only the homeopaths complaining here about junk science? Hmmmm.

  5. apgaylard said

    Here, commenting on this, as well.

  6. pleick said

    An important post, as the study by Shang is often criticised for its “secrecy”.

    In one of your previous posts, you did some research about trials often cited that, apparently, prove homeopathy’s effectiveness. The results really are disheartening. Time and again, in alternative journals, you have to check the sources and references. Because more often than not, they don’t say what the author claims or implicates that they say.
    One of the things that make wading through these articles so tedious…

    The Lancet article by Shang et al. is a nice counter example: a quick search reveals the necessary information.

  7. apgaylard said

    pleick:
    Thanks for your comments and taking the trouble to read my posts.

    Your comment is spot on; unfortunately it’s not just the ‘altie’ literature that shows a distinct lack of integrity with referencing.

    I read a terrible example in a paper that I refereed for a major automotive engineering conference last year: the reference said the exact opposite of what the author said it said!

  8. pleick said

    True, it’s not just the “alties”. Now your last comment does worry me a bit! Like you, I’m a physicist/engineer, I work in the automotive industry and publish papers in automotive conferences…

    I do know that referencing is often far from perfect – even in serious disciplines or good journals. But honestly, if I were to referee a paper, I’d generally take the references on trust – unless I already knew the source or if it was very important to check exactly what the source says. This may be asking too much, but correct referencing seems just like correct spelling: a basic skill. Of course there will be the occasional typo (just check my posts!), but a referee should look at more important aspects than basic technical details.

    Of course, having read quite a few papers on “fringe” topics… my experience is that THERE, checking the references is of cardinal importance. To sum it up: in alternative journals, concerning bad referencing, it’s “look and you shall find”. It doesn’t even take too much effort… In my own field, my limited experience tells me something else: there is the exceptional truely dishonest quotation, but mostly, some slight exagerations, a little bit of cherry picking… not the highest standards, just normal human character flaws…

  9. apgaylard said

    pleick:

    I agree, in ‘altie’ world one has to check everything. In my professional role I have got into the habit of checking references when I referee.

    However, my biggest complaint is ususally the absence of appropriate references. To appropriate a term from Kuhn, “normal science” is an incremental process; it’s vital to set work in an appropriate context. In my experience an increasing number of authors fail to do this.

Sorry, the comment form is closed at this time.

 
Follow

Get every new post delivered to your Inbox.

Join 595 other followers

%d bloggers like this: