Publish and be damned

One of the things that I’ve noted as I have been drawn into debates with advocates of homeopathy, dowsing, and other strange belief systems over the last year or so is the total lack of appreciation for what science is and how it works.  Its values and methods are misunderstood, misrepresented, misinterpreted, or even maligned.

This week a homeopath, Clive Stewart, has provided an excellent example of the failure to understand the value of scientific publications and the debates they provoke.  I’ve been exchanging views with him on the merits of the meta-regression analysis of homeopathic treatments published by Shang et al in The Lancet.  He made some claims about the ailments covered by the eight ‘higher quality’ – lowest bias studies of homeopathy from which the authors concluded, “the effects seen in placebo controlled trials of homoeopathy are compatible with the placebo hypothesis”.  I demonstrated that these claims were incorrect and he went on to make the very guarded apology shown below.

“The 8 studies I quoted were from a commentary of the paper published by the European Committee for Homeopathy.  The information was quoted as being given by individuals who were involved in the study itself.  It can neither be disproved or verified.

We only have the word of Shang et al that the “final 8″ you have cited were indeed the original studies that were used to conclude the meta analysis.  In the light of the lead authors admission of bias and proven lack of transparency I would still treat this information with suspicion.  However, as the information you have quoted was from the authors themselves it was unfair to call your citings incorrect and I apologise.”

Comment on Homeopathy – good news, Posted by Clive Stuart, August 12th, 2008 at 12:50 pm

One of the fascinating things about this statement is that it shows a total lack of understanding of the role of the open literature in the development of scientific ideas.  When a scientist publishes a paper they put their ideas, data, and analyses in front of the wider scientific community for scrutiny.  By the time they have got to this stage they have usually had their work critiqued by colleagues during informal discussions, followed by participants at colloquia and conferences.  Then, getting published in a good journal requires passing further reviews by expert referees.  However, this doesn’t mean that the work is now beyond criticism and revision; far from it: this moves the work on to the next stage of scrutiny.  A wider audience can now assess the work; those with criticisms to level often get letters published in the same journal, to which the authors reply.

This is, to be fair, an imperfect and variable process.  However, it does provide some mechanisms for correction and debate.  Even if an author rashly dismisses published criticism; it’s still there and readers can make their own judgements.

In the case of Shang et al there were a number of critical letters published in The Lancet: Fisher et al., Linde  and Jonas, Walach et al., and Dantas.  Unlike most of the author’s replies published by the journal Homeopathy in the wake of its ‘memory of water’ issue; Shang and his co-authors directly addressed the criticisms and in one crucial respect refined their publication.

Critics pointed out that they had not identified the final eight studies of homeopathy and six studies of conventional medicine upon which they based their view of the efficacy of homeopathy.  So in their letter the authors identified the contentious studies and made a list of all the studies included in the analysis available as an appendix to the on-line version of the paper.  Finally, they made available lists of the homoeopathy studies they excluded from their analysis (along with their reasons); details of the key characteristics and results of all the homoeopathy trials that they did include, along with the comparable data for the conventional trials they used as matched controls.

This is how science usually works (unless the ‘science by press release’ gambit epitomized by Pons and Fleischmann is deployed): published studies are read and criticized; the authors either address the points raised or the credibility of their work is diminished.  In this case we have authors who were prepared to engage positively with their critics and make further data available.

However, this is not good enough for our homeopath.  I get the impression that because this process took four whole months, in his world the authors are guilty of a, “proven lack of transparency”.  Neither does he understand the significance of the data that has been in the public domain for nearly three years, “We only have the word of Shang et al that the “final 8″ you have cited were indeed the original studies that were used to conclude the meta analysis”.

No, we don’t.  Sufficient data is now in the public domain for anyone with the skills and inclination to replicate the study.  This is another vital function of a scientific publication.  It may not be glamorous work and it may not happen often enough; but the scientific community can repeat the work and see if the original authors were right. (The attempted replications of Pons and Fleischmann’s so-called cold fusion experiment are good examples of its value.)

Also, the possibility that others can attempt to replicate your work is an incentive to be honest in the first place.  Just imagine if Shang et al had based their conclusions on one set of studies and then claimed that they had used another:  they would have provided an alarming hostage to fortune.  Just think of the fuss that homeopaths could make if they could prove fraud in this case.  The careers of Shang and his fellow authors would probably be over – their reputations would certainly be in tatters.  There are not many who would risk such a thing; particularly when they could be sure that their conclusion would upset such a vociferous community.  This would be sheer madness.

The fact that, to my knowledge, no one has said that they have failed to replicate the results makes me think that the authors were telling the truth.  (According to homeopath Dana Ullman an alternative analysis of the studies looked at by Shang et al, plus some more that critics felt should have made the cut is soon to be published in the Journal of Clinical Epidemiology. – Interview with Dana Ullman by Louise Mclean, Sue Young Homeopathy, 14th August 2008.)

As this study was an exercise in analyzing data that any interested person could get their hands on; publishing the identity of the studies is akin to opening up your lab book to public scrutiny.  This resonates with a suggestion made in Nature for curbing scientific fraud, reported on by Dan Greenberg, “The Nature authors briefly refer to the most useful suggestion yet made in the long-running concerns over scientific misconduct: Occasional audits of laboratory records to match up publications with the underlying research.  Knowing that the auditors may drop in unannounced could have a wondrously elevating influence on scientific integrity.”

In effect, this is the position that Shang and his co-authors have put themselves in.  This, I think, speaks well of their scientific integrity.

So when Stuart claims that, “The 8 studies I quoted were from a commentary of the paper published by the European Committee for Homeopathy.  The information was quoted as being given by individuals who were involved in the study itself.  It can neither be disproved or verified” he is wrong.  A replication of the analysis of Shang et al would do nicely.

This moves us into another area that True Believers in strange things struggle to cope with: telling the difference between reliable and unreliable sources. 

Real scientific communities have key publications and events that draw on the community for expert reviewers, who act as gate-keepers by the application of shared scientific values.  Reputable, peer-reviewed journals are usually the main source of reliable information.  Depending on the community, conference papers are either a trusted source of data or a testing-ground for new ideas.

This body of knowledge is, of course, imperfect: reliant on the quality of the contributors, editors, reviewers, and readers.  It is also dynamic: papers come under criticism; author’s respond; sometimes errata are published and, in extreme cases, papers can be withdrawn.  At its best, it provides a fierce crucible that tests and refines.

My observation is that for communities of True Believers, the approach is very different.  Testimonials and customer satisfaction surveys are hailed as hard evidence.  Really silly books are cited.  Pseudo-journals controlled by cabals of other True Believers publish obvious nonsense: the many elementary errors in the ‘memory of water‘ issue of Homeopathy and the quantum flap-doodle published in eCAM provide eloquent confirmation of this point.

When it comes to published studies that they like, the attitude is frequently one of supine deference to the written word:  they are celebrated not critiqued.  Limitations and errors are ignored and the paper is used over and over again to justify belief.  (Continued references to Beneviste‘s debunked Nature paper or the flawed COPD study reported in Chest are good examples of the genre.)  It will be interesting to see the degree of scrutiny that the forthcoming Journal of Clinical Epidemiology receives from homeopathic apologists.  Will it be held to the high standards demanded of Shang et al?

This is may explain why Stuart continues to show such trust in a self-published report from a homeopathy advocacy group containing little more than hearsay provided by anonymous individuals.  In his homeopathic fantasy it is the corresponding author of Shang et al who is biased and their transparent information which is to be treated with suspicion.

The inability of homeopathy advocates to understand the role of the open literature, the process of testing and revising ideas, along what reliable sources look like, are yet more reasons why ‘homeopathic science’ is an oxymoron.

[This post is an elaboration of some ideas I covered in a comment I posted on Homeopathy – good news (August 12th, 2008 at 1740)]

Advertisement