A canna’ change the laws of physics

Scotty, The Naked Time, stardate 1704.3, Episode 7

Publish and be damned

Posted by apgaylard on August 16, 2008

One of the things that I’ve noted as I have been drawn into debates with advocates of homeopathy, dowsing, and other strange belief systems over the last year or so is the total lack of appreciation for what science is and how it works.  Its values and methods are misunderstood, misrepresented, misinterpreted, or even maligned.

This week a homeopath, Clive Stewart, has provided an excellent example of the failure to understand the value of scientific publications and the debates they provoke.  I’ve been exchanging views with him on the merits of the meta-regression analysis of homeopathic treatments published by Shang et al in The Lancet.  He made some claims about the ailments covered by the eight ‘higher quality’ – lowest bias studies of homeopathy from which the authors concluded, “the effects seen in placebo controlled trials of homoeopathy are compatible with the placebo hypothesis”.  I demonstrated that these claims were incorrect and he went on to make the very guarded apology shown below.

“The 8 studies I quoted were from a commentary of the paper published by the European Committee for Homeopathy.  The information was quoted as being given by individuals who were involved in the study itself.  It can neither be disproved or verified.

We only have the word of Shang et al that the “final 8″ you have cited were indeed the original studies that were used to conclude the meta analysis.  In the light of the lead authors admission of bias and proven lack of transparency I would still treat this information with suspicion.  However, as the information you have quoted was from the authors themselves it was unfair to call your citings incorrect and I apologise.”

Comment on Homeopathy – good news, Posted by Clive Stuart, August 12th, 2008 at 12:50 pm

One of the fascinating things about this statement is that it shows a total lack of understanding of the role of the open literature in the development of scientific ideas.  When a scientist publishes a paper they put their ideas, data, and analyses in front of the wider scientific community for scrutiny.  By the time they have got to this stage they have usually had their work critiqued by colleagues during informal discussions, followed by participants at colloquia and conferences.  Then, getting published in a good journal requires passing further reviews by expert referees.  However, this doesn’t mean that the work is now beyond criticism and revision; far from it: this moves the work on to the next stage of scrutiny.  A wider audience can now assess the work; those with criticisms to level often get letters published in the same journal, to which the authors reply.

This is, to be fair, an imperfect and variable process.  However, it does provide some mechanisms for correction and debate.  Even if an author rashly dismisses published criticism; it’s still there and readers can make their own judgements.

In the case of Shang et al there were a number of critical letters published in The Lancet: Fisher et al., Linde  and Jonas, Walach et al., and Dantas.  Unlike most of the author’s replies published by the journal Homeopathy in the wake of its ‘memory of water’ issue; Shang and his co-authors directly addressed the criticisms and in one crucial respect refined their publication.

Critics pointed out that they had not identified the final eight studies of homeopathy and six studies of conventional medicine upon which they based their view of the efficacy of homeopathy.  So in their letter the authors identified the contentious studies and made a list of all the studies included in the analysis available as an appendix to the on-line version of the paper.  Finally, they made available lists of the homoeopathy studies they excluded from their analysis (along with their reasons); details of the key characteristics and results of all the homoeopathy trials that they did include, along with the comparable data for the conventional trials they used as matched controls.

This is how science usually works (unless the ‘science by press release’ gambit epitomized by Pons and Fleischmann is deployed): published studies are read and criticized; the authors either address the points raised or the credibility of their work is diminished.  In this case we have authors who were prepared to engage positively with their critics and make further data available.

However, this is not good enough for our homeopath.  I get the impression that because this process took four whole months, in his world the authors are guilty of a, “proven lack of transparency”.  Neither does he understand the significance of the data that has been in the public domain for nearly three years, “We only have the word of Shang et al that the “final 8″ you have cited were indeed the original studies that were used to conclude the meta analysis”.

No, we don’t.  Sufficient data is now in the public domain for anyone with the skills and inclination to replicate the study.  This is another vital function of a scientific publication.  It may not be glamorous work and it may not happen often enough; but the scientific community can repeat the work and see if the original authors were right. (The attempted replications of Pons and Fleischmann’s so-called cold fusion experiment are good examples of its value.)

Also, the possibility that others can attempt to replicate your work is an incentive to be honest in the first place.  Just imagine if Shang et al had based their conclusions on one set of studies and then claimed that they had used another:  they would have provided an alarming hostage to fortune.  Just think of the fuss that homeopaths could make if they could prove fraud in this case.  The careers of Shang and his fellow authors would probably be over – their reputations would certainly be in tatters.  There are not many who would risk such a thing; particularly when they could be sure that their conclusion would upset such a vociferous community.  This would be sheer madness.

The fact that, to my knowledge, no one has said that they have failed to replicate the results makes me think that the authors were telling the truth.  (According to homeopath Dana Ullman an alternative analysis of the studies looked at by Shang et al, plus some more that critics felt should have made the cut is soon to be published in the Journal of Clinical Epidemiology. – Interview with Dana Ullman by Louise Mclean, Sue Young Homeopathy, 14th August 2008.)

As this study was an exercise in analyzing data that any interested person could get their hands on; publishing the identity of the studies is akin to opening up your lab book to public scrutiny.  This resonates with a suggestion made in Nature for curbing scientific fraud, reported on by Dan Greenberg, “The Nature authors briefly refer to the most useful suggestion yet made in the long-running concerns over scientific misconduct: Occasional audits of laboratory records to match up publications with the underlying research.  Knowing that the auditors may drop in unannounced could have a wondrously elevating influence on scientific integrity.”

In effect, this is the position that Shang and his co-authors have put themselves in.  This, I think, speaks well of their scientific integrity.

So when Stuart claims that, “The 8 studies I quoted were from a commentary of the paper published by the European Committee for Homeopathy.  The information was quoted as being given by individuals who were involved in the study itself.  It can neither be disproved or verified” he is wrong.  A replication of the analysis of Shang et al would do nicely.

This moves us into another area that True Believers in strange things struggle to cope with: telling the difference between reliable and unreliable sources. 

Real scientific communities have key publications and events that draw on the community for expert reviewers, who act as gate-keepers by the application of shared scientific values.  Reputable, peer-reviewed journals are usually the main source of reliable information.  Depending on the community, conference papers are either a trusted source of data or a testing-ground for new ideas.

This body of knowledge is, of course, imperfect: reliant on the quality of the contributors, editors, reviewers, and readers.  It is also dynamic: papers come under criticism; author’s respond; sometimes errata are published and, in extreme cases, papers can be withdrawn.  At its best, it provides a fierce crucible that tests and refines.

My observation is that for communities of True Believers, the approach is very different.  Testimonials and customer satisfaction surveys are hailed as hard evidence.  Really silly books are cited.  Pseudo-journals controlled by cabals of other True Believers publish obvious nonsense: the many elementary errors in the ‘memory of water‘ issue of Homeopathy and the quantum flap-doodle published in eCAM provide eloquent confirmation of this point.

When it comes to published studies that they like, the attitude is frequently one of supine deference to the written word:  they are celebrated not critiqued.  Limitations and errors are ignored and the paper is used over and over again to justify belief.  (Continued references to Beneviste‘s debunked Nature paper or the flawed COPD study reported in Chest are good examples of the genre.)  It will be interesting to see the degree of scrutiny that the forthcoming Journal of Clinical Epidemiology receives from homeopathic apologists.  Will it be held to the high standards demanded of Shang et al?

This is may explain why Stuart continues to show such trust in a self-published report from a homeopathy advocacy group containing little more than hearsay provided by anonymous individuals.  In his homeopathic fantasy it is the corresponding author of Shang et al who is biased and their transparent information which is to be treated with suspicion.

The inability of homeopathy advocates to understand the role of the open literature, the process of testing and revising ideas, along what reliable sources look like, are yet more reasons why ‘homeopathic science’ is an oxymoron.

[This post is an elaboration of some ideas I covered in a comment I posted on Homeopathy – good news (August 12th, 2008 at 1740)]

10 Responses to “Publish and be damned”

  1. dvnutrix said

    And a fine elaboration it is. It is encouraging that people are going to produce their own analysis but it is always irritating that some people are so dismissive of systematic reviews that plainly cite criteria for inclusion and exclusion of particular studies. “I like this study’s outcomes because it is in line with my personal preferences” should not be sufficient.

    We only have the word of Shang et al that the “final 8″ you have cited were indeed the original studies that were used to conclude the meta analysis. In the light of the lead authors admission of bias and proven lack of transparency I would still treat this information with suspicion.

    That has to be an all-time non-apology. If there were a competition then I would nominate this.

  2. apgaylard said

    dvnutrix: Thanks for the kind comment. I quite liked his assertion that the exclusion criteria were somehow mysterious – when, as you say, they are included very clearly in the paper. I think that believing in homeopathy requires abandonment (or ignorance) of scientific values to the point where its advocates just can’t engage properly with scientific processes.

  3. draust said

    Nice post – agree completely. I have been musing on some similar themes (though a lot less coherently) over on the blog here.

  4. draust said

    PS This also highlights one reason (among many) why it is utterly laughable purporting to have degrees in CAM that are “scientific” – as stressed by all the scientists commenting in the THE here

  5. apgaylard said

    draust: Thanks doc; you are too modest though. I read your post with great interest. The UCLAN situation is certainly deplorable. The more I look into the values of the alt.med community, the greater the gulf I see between it and anything I would recognise as science.

  6. draust said

    The more I look into the values of the alt.med community, the greater the gulf I see between it and anything I would recognise as science.

    Yes, I would agree with that, Adrian. When I started writing about AltMed themes a couple of years back, I was much more sympathetic to CAM, notably the less outlandish things like herbalism and acupuncture, than is/was David Colquhoun, with whom I exchanged the odd email on the subject. But after two years dealing with them on the blogs, I am pretty much aligned with where David is.
    The more one debates them, the more clear it becomes that most of the CAM people cannot, will not, or dare not (take your pick) grasp the differences between science (as scientists understand it) and cargo cult science (as practised by the CAM lot).

    Of course, you could perhaps say that some of what they are missing is the “tacit knowledge” (in the Harry Collins sense) about how science is done and works that scientists acquire through their training and practise… …but in fact even the Woo-sers who have been trained as conventional scientists or doctors, like George Lewith and Lionel Milgrom, seem wholly to abandon all adherence to “scientific ways of finding out” when they embrace Alternative Reality.

  7. […] Although not about chiropractic, we would recommend these glimpses into Alt.Reality ™ and Alt.Science by Dr Aust and Gaylard. One alternative reality please. No – make that several and Publish and be damned. […]

  8. pleick said

    So when Stuart claims that, “The 8 studies I quoted were from a commentary of the paper published by the European Committee for Homeopathy. The information was quoted as being given by individuals who were involved in the study itself. It can neither be disproved or verified” he is wrong. A replication of the analysis of Shang et al would do nicely.

    … and it reminds you of the last resort exit strategy: the data can’t be trusted anyway, since the studies are being financed by “big pharma”, which has to supress knowledge about cheap, alternative methods to keep it’s profits flowing…
    It’s a really nice immunisation strategy, and it makes it’s adherents even feel better about themselves, since they can claim to be part of some “enlightened elite”.

    Personally, I wouldn’t trust any private company with financial interests that much… but of course, luckily, medical research is not the monolithic block that some “alties” perceive it to be. Perhaps the fact that the whole of the scientific community doesn’t take them seriously fosters that delusion.

  9. jdc325 said

    Re P Leick’s comment: “It’s a really nice immunisation strategy, and it makes it’s adherents even feel better about themselves, since they can claim to be part of some “enlightened elite”.”
    There does seem to be a strand running through alternative medicine that shows some ‘true believers’ to think of themselves as enlightened and of those following the mainstream or conventional to be either part of the big conspiracy – or ‘sheep’ mindlessly following those who would lead them astray. The same probably goes for 9/11 truthers and those who believe in the Illuminati, lizard rulers etc etc.

    Thanks for this post Adrian – enjoyed reading it.

  10. […] Nailati on Sat 20-12-2008 3 Ways to Increase Web Rankings Saved by Yourong18 on Sat 20-12-2008 Publish and be damned Saved by pip on Thu 18-12-2008 PRESIDENTIAL DOODLES: Scribbles, Scratches, Squiggles, and Scrawls […]

Sorry, the comment form is closed at this time.

 
%d bloggers like this: