A canna’ change the laws of physics

Scotty, The Naked Time, stardate 1704.3, Episode 7

The Myth of The Secret Eight

Posted by apgaylard on November 28, 2007

There seems to be a persistent myth among homeopathic apologists that the final eight homeopathic studies (large and of high quality) analysed in the famous Lancet paper by Shang et al. are somehow secret.

Here is a recent example of this piece of homeomythology:

“… When Shang et al (Lancet 2005) did a metaanalysis of 110 homeopathy trials, homeopathy was significantly better than placebo. When they restricted it to “high quality” trials (n=20), homeopathy was still significantly better than placebo, same as the ‘matched’ biomedical trials. They then restricted it to “large, high quality” trials (n=8), and claimed they were no better than placebo. Does anyone here know which 8 homeopathy RCTs were selected? Failing that, the 20 “high quality” trials, from which one could identify the 8 largest. And how and when did they set the cutpoint for “large”?”

“Chamfort”, Comment is Free, Guardian, Comment No. 941849, November 22 13:11

So, is it possible to find which eight trials of homeopathy were finally selected?  Yes.  In fact one poster did provide an answer to this question.  He noted that “…The 8 homeopathy trials and 6 conventional trials that were considered to be large, high quality trials are identified by the authors in their reply to comments on the original article. See Lancet 366 (2005); p. 2083. The list of included trials can be found as an appendix to the online version of the original article, see Lancet 366 (2005); 726-732.”

Where did this idea come from?  Well, Shang et al. did omit these details from their original paper.  This was picked up by many apologists, including correspondents to the Lancet: Fisher et al., Linde  and Jonas, Walach et al., and Dantas.  Shang et al. then provided the information, as noted above, in the same issue of the Lancet as the critical letters.

But, as we have seen, the complete story has been missed by some homeopathic apologists.  It would seem that they only read the critical correspondence, or quotations from it.  This ‘secrecy’ myth has now passed into homeopathic folklore.

Here’s a very recent comment from a pro-homeopathy blog:

“…Another oft-raised issue is that homeopathy cannot stand up to conventional medical trials, and that homeopathic studies fail meta-analysis. It is suggested that homeopaths “cherry pick” the results of studies to confirm positive results. Iris Bell MD Phd, director of research at the Program in Integrative Medicine at the University of Arizona College of Medicine, stated the following in response to a negatively biased article in the British medical journal “The Lancet” in August of 2005;

“The researchers started with 110 homeopathy studies and 110 conventional medicine studies, but drew their main conclusions from 8 homeopathy and 6 conventional studies. And in an odd decision by the journal and the researchers, those 14 studies were not identified. This makes it impossible for others to challenge or confirm their conclusions. Given the tiny number left, they really were under an obligation to tell the reader which they used.” …”

http://goodscience.wordpress.com, 24th November 2007.

This quotes an article that Bell had published in the Journal Of Alternative And Complementary Medicine before the additional information was made available by Shang et al.

So, this criticism gets made over and over again; every time the meta-analysis of Shang et al. is discussed.  It makes me wonder why.  Perhaps perceiving themselves as upholders of an unchanging tradition, homeopathic apologists don’t think that the argument moves on?  Maybe a paranoid world view that includes the great allopathic conspiracy makes them think that such omissions are made with malign intent and evil scientists won’t correct them?  Perhaps it’s just sloppy scholarship?

In fact, quite a lot of detail about the studies included and excluded by Shang et al. is freely available online.  On the website of Shang’s home institution, Institut für Sozial und Präventivmedizin at Universität Bern, there are three files that can be downloaded:

“… List of excluded homoeopathy studies (PDF 32KB) provides a list of excluded homoeopathy studies with reasons for exclusions.

Study characteristics of homoeopathy studies (PDF 72KB) provides details on study characteristics and results of 110 placebo-controlled homoeopathy trials.

Study characteristics of allopathy studies (PDF 72KB) provides details on study characteristics and results of 110 placebo-controlled allopathy trials. …”

This information was originally posted on December 17th, 2005 and subsequently corrected (for an error in the graphs) on December 23rd, 2005.

It’s easy to use the second file to determine the identity of the ‘secret eight’:  select the trials identified as of high quality, list them by size and pick the eight largest.  The results of doing this are shown in Figure 1.  The ‘secret eight’ are obvious; as is the cut-off criterion.

The Secret 8 Revealed

Figure 1  The Secret Eight Revealed

Table 1 provides details of the intervention used, clinical indication and outcome.  Two of the ‘secret eight’ studies were of individualised homeopathy.  Others used single or multiple preparations and two were of proprietary homeopathic ‘medicines’.

High Quality Study





Rottey (80)


Mucococcinum 200K

Influenza like disease

Occurrence of symptoms

Vickers (94)


Arnica 30X

Physical activity


Papp (71)



Influenza like disease

Global assessment, patient

Schmidt (84)


Thyroidinum 30C


body weight

Labreque (55)


Thuya 30C, Ant 7C, Nitr acid 7C


Global assessment, physician

Jacobs (46)


Individual treatment


Duration of diarrhoea

Weiser (97)


Euphorbium compositum®


Global assessment, other

Walach (96)


Individual treatment



Jacobs (49)


Individual treatment, 30C


Duration of diarrhoea

Jacobs (47)


Individual treatment

Otitis media

Global assessment, other

Hart (41)


Arnica C30



Wiesenauer (102)


Galphimia glauca D4


Global assessment, other

Zell,von, (105)



Sprain Joint


Böhmer (15)



Sports injury

Global assessment, patient

Vickers (95)


Arnica 30C, Rhus t 30C, Sarc a 30C

Physical activity


Chapman (21)


Individual treatment

Traumatic brain injury


Jawara (50)


Arnica 30C, Rhus t 30C

Physical activity


Tveiten (91)


Arnica D30

Physical activity


Stevinson (86)


Arnica C6

Hand surgery


Lepaisant (58)


Folliculinum 9C

Premenstrual syndrome

Global assessment, other

Chapman (22)


Individual treatment

Premenstrual syndrome

Global assessment, other

Table 1.  Details of the High Quality Homeopathic Studies Identified by Shang et al

The use of single preparations, or set ‘medicines’, in clinical trials is often criticised by homeopaths; however they seem happy enough when the results appear to be  positive.

I have also yet to see evidence of concerted criticism from the homeopathic community of high street retailers like Boots selling ‘off the shelf’ homeopathic preparations for treating particular conditions.

Homeopathic apologists do tend to miss the point of Shang et al.  This analysis demonstrates (again) that removing bias, trials of low quality and small size progressively diminishes the apparent clinical effects of homeopathy to the point where they cannot be distinguished from a placebo.  This is just what would be expected if homeopathy was just a placebo.  Any apparent effects in excess of this are just so much statistical noise.

So, finally, let’s dispel this particular piece of homeomythology:  the identity of the final eight, high quality relatively large, homeopathic trials selected by Shang et al. are not secret.  They have been in the public domain for nearly two years.

9 Responses to “The Myth of The Secret Eight”

  1. Nice digging.

  2. wilsontown said

    Excellent stuff. Was going to write something on this myself, but now I think I needn’t bother.

    I’ve yet to see a critique of the Shang et al. meta-analysis that stands up to scrutiny, and I’ve been looking.

  3. apgaylard said

    Wilsontown: Thanks. This one has been on my mind for a while. I must have dowloaded the PDF files over a year ago. Thanks for your CIF post; it galvanised me into action!

  4. Not a moment too soon, because here is Dana Ullman:

    Perhaps, SOMEONE can finally tell us which were the 21 homeopathic trials and the 9 allopathic ones. Shang NEVER divulged, most likely because this review would show real benefits from homeopathic treatment. Isn’t anyone suspicious of “black box” comparison studies like this? Why are only the homeopaths complaining here about junk science? Hmmmm.

  5. apgaylard said

    Here, commenting on this, as well.

  6. pleick said

    An important post, as the study by Shang is often criticised for its “secrecy”.

    In one of your previous posts, you did some research about trials often cited that, apparently, prove homeopathy’s effectiveness. The results really are disheartening. Time and again, in alternative journals, you have to check the sources and references. Because more often than not, they don’t say what the author claims or implicates that they say.
    One of the things that make wading through these articles so tedious…

    The Lancet article by Shang et al. is a nice counter example: a quick search reveals the necessary information.

  7. apgaylard said

    Thanks for your comments and taking the trouble to read my posts.

    Your comment is spot on; unfortunately it’s not just the ‘altie’ literature that shows a distinct lack of integrity with referencing.

    I read a terrible example in a paper that I refereed for a major automotive engineering conference last year: the reference said the exact opposite of what the author said it said!

  8. pleick said

    True, it’s not just the “alties”. Now your last comment does worry me a bit! Like you, I’m a physicist/engineer, I work in the automotive industry and publish papers in automotive conferences…

    I do know that referencing is often far from perfect – even in serious disciplines or good journals. But honestly, if I were to referee a paper, I’d generally take the references on trust – unless I already knew the source or if it was very important to check exactly what the source says. This may be asking too much, but correct referencing seems just like correct spelling: a basic skill. Of course there will be the occasional typo (just check my posts!), but a referee should look at more important aspects than basic technical details.

    Of course, having read quite a few papers on “fringe” topics… my experience is that THERE, checking the references is of cardinal importance. To sum it up: in alternative journals, concerning bad referencing, it’s “look and you shall find”. It doesn’t even take too much effort… In my own field, my limited experience tells me something else: there is the exceptional truely dishonest quotation, but mostly, some slight exagerations, a little bit of cherry picking… not the highest standards, just normal human character flaws…

  9. apgaylard said


    I agree, in ‘altie’ world one has to check everything. In my professional role I have got into the habit of checking references when I referee.

    However, my biggest complaint is ususally the absence of appropriate references. To appropriate a term from Kuhn, “normal science” is an incremental process; it’s vital to set work in an appropriate context. In my experience an increasing number of authors fail to do this.

Sorry, the comment form is closed at this time.


Get every new post delivered to your Inbox.

Join 615 other followers

%d bloggers like this: