Blind to the Rudolph Effect

A recent discussion on the bad science forum [thanks to Bryn for highlighting this topic] set me thinking about the claims made for the treatment of hay fever by red light phototherapy.

If you’ve not come across these claims before, they can be seen in the advertising material provided on-line by Lloyds Pharmacy for their “Allergy Reliever” and Health Innovations for their “Medinose Hayfever Treatment“.

These are similar devices that allow you to shove some light-emitting diodes (LED) up your nose and administer a ‘dose’ of red light to the nasal mucosa.  What is the proposed mechanism of action?  Health Innovations say, “Medinose inhibits the release of histamine, relieving or even completely eliminating allergic reactions and complaints in a natural way.”  Lloyds make a similar claim, “Allergy Reliever uses red light therapy to suppress the cells that release histamine, thereby relieving the symptoms of hay fever. ” 

What is the evidence?  Lloyds Pharmacy claims that their device is, “Clinically proven to relieve allergy symptoms.”  This is a very strong advertising claim and so we should expect to find commensurately strong evidence.

Health Innovations don’t make the same claim, but they do say, “The Medinose has undergone extensive clinical testing, with over 70% of patients reporting a significant reduction in allergic symptoms.”

When I queried the evidence with them they very kindly provided me with a copy of the study they refer to on their website: Nueman, I. and Finkelstein, Y. “Narrow-band red light phototherapy in perennial allergic rhinitis and nasal polyposis” Annals of Allergy, Asthma & Immunology, 1997; 78:399-406. (copies are available on-line here and here)

This was a, “double-blind randomized prospective study” that included, “50 patients with allergic rhinitis and 10 with nasal polyposis“.  They were given intranasal illumination with red light (660 nm) of 4.4 minutes three times per day; this gave a total dose of 6 joule/day.  The study ran for fourteen days.

A control group of twenty-nine of the patients with rhinitis and one with polyposis received what the authors called “sham illumination” as a placebo.  Improvement of symptoms was self-reported by 72% of the patients with rhinitis in the treatment group and by 24% in the placebo group.  The authors also sought to evaluate any improvement endoscopically, and reported improvements in 70% and 3% of the subjects in the two groups, respectively.

No improvement was seen in any of the ten subjects with polyposis.

A search of PubMed reveals that this is the only indexed paper on the use of red light phototherapy for this condition (UV phototherapy seems to be more popular, with one group at least – [Koreck et al, 2007, Koreck et al, 2005(1) , Koreck et al, 2005(2)]).

This seems to fall well short of what would be expected for a clinically proven intervention.  Also Health Innovations claim of, “extensive clinical testing” seems to be based on a single study – the 70% figure would appear to come from the Nueman and Finkelstein paper.

So, how good is this pivotal study?  To get an overall view I thought that I’d score it using R. Barker Bausell’s approach.  As it is a study published in a peer-reviewed journal, it passes the first test. 

Next, Bausell recommends approaching the article with ‘extreme scepticism’; it’s good to bear in mind that this is a core scientific value, virtue even.  Then he recommends giving, “more credence to trials published in well known medical journals”.  Annals of Allergy, Asthma & Immunology appears to be a respectable, but not top-of-the-line journal (it had an impact factor of 2.254 in 2006).

Next Baussel counsels, “Give more credence to trials conducted in English and Scandinavian language speaking countries. A systematic review published several years ago found that investigators from certain countries (most notably China and Russia ) produced only positive results in their acupuncture trials.” 

This was work conducted in Israel; it’s unclear how this should be weighted – if at all.  However, it is interesting to note that Baussel cites the results obtained by Vickers et al published in their 1998 article “Do Certain Countries Produce Only Positive Results? A Systematic Review of Controlled Trials”.  This review of 252 published acupuncture trials contained only one from Israel; it was positive.  There’s not enough evidence here to go on – so I’ll give Nueman and Finkelstein a pass on this point.

Next Bausell invites us to see whether there are, “statistically significant (or reliable or clinically significant) differences between the placebo (or sham) group and the treatment group”.  And it appears that there are.  Table 1, below, shows the p values obtained for the eleven variables measured in the study.

Table 1 – p values for each study variable

Study Variable	P Value
Subjective
Nasal obstruction	0.016
Headache	0.023
Nasal Itching	0.019
Cough	0.004
Rhinorrhea	0.0004
Overall (subjective) improvement	0.021
Objective
Appearance of nasal mucosa	0.017
Ethmoidal edema	0.0007
Nasal passage	0.0002
Nasal discharge	0.022
Overall (Objective) improvement	0.0006

All of these variables show statistically significant differences between the treatment and placebo group at the 5% (p=0.05) level.  However, if the significance level were to be adjusted to account for the fact that eleven variables were measured for each participant, then the number of statistically significant results would reduce.  For example, applying the Bonferroni correction would imply that the statistical significance level should be set at 0.05/11, or 0.0045.  This would reduce the statistically significant outcomes to the subjective improvements in cough and rhinorrhea along with objective measures of ethmoidal edema, nasal passage and overall improvement.

However, we do still have some statistically significant improvements.  Were these clinically significant?  As ‘marked’ improvement was noted in 29% of the subjects with allergic rhinitis who received treatment, then perhaps we also have some clinically significant improvement.

The next issues raised by Bausell get to the crux of the matter.  Critically, this starts by inviting us to ask, “Did the trial employ a randomly assigned placebo-sham control group indistinguishable from the therapy being evaluated?”

The placebo, or sham, therapy is described in the paper as the treatment device (BioNase), “with internally disconnected light emitting diodes”.  Simply, the sham illumination consists of no illumination.  Is this important?  Just consider some comments made by happy Medinose purchasers on Amazon.

“I haven’t had the guts to use it in front of my kids as I don’t think I could take the ‘red nose jokes’ as the lights do shine through brightly.”

 “When using the medinose it does look like rudolph the reindeer has arrived …”

“…when you have this switched on, you look hilarious! It makes your whole nose glow red, which I hadn’t realised (I wondered why my father-in-law started singing “Rudolph the Red Nosed Reindeer” the first time I used it!)”

Unless someone using this device has a very short nose, they are bound to see it glow red if they are getting the treatment (it seems one of the Amazon customers falls into this category).  Those not receiving the treatment will not see the ‘rudolph effect’ at all.  This is very likely to break the blinding for the patient and certainly for any experimenter administering the treatment. (If the treatment is self-administered at home, then the reaction of the patients family and friends will cetrainly break the blinding.) Also, if I was a participant the first thing that I’d do would be to switch the thing on and look at the light before inserting it into my nose. I’d be curious. If I was in the placebo group I’d know it right away. I doubt that I’m unusual in that respect.

The sham non-illumination is not indistinguishable from the therapy.  This is a fatal flaw.  It would have been far better had a different type of light been used.  The published work on UV phototherapy for allergic rhinitis seems to use a white light placebo; so the use of non-illumination appears to be a major oversight.

If the authors didn’t want to use placebo illumination, then why not incorporate a nose-shroud into the design?  Either way, a suitable sham treatment does not seem hard to come up with.  That the authors failed to do so shows that this work was not well executed.

The next major problem with this study is its size. Bausell asks, “Did the trial employ at least 50 participants per group?”  The answer here is no.  The rhinitis treatment group did include 50 participants, but the relevant control group had only 29.  Even worse, the polyposis treatment group had only 10 participants and a single ‘control’. 

In terms of allergic rhinitis, this is a small study.  It’s hard to see how any conclusions can be drawn about the effect on polyposis.  At best, with a proper placebo, this work could have shown whether a larger study of the use of red light phototherapy for allergic rhinitis is warranted.  It cannot sustain claims of clinical proof.

The final point to check is, “Did 25% or more of the participants drop out of the study before it was over?”  No information is provided on drop-outs, so we have to assume that everyone completed the fourteen day course.  Given that this is a short period, this is probably not unreasonable.

So, what we are left with is a single, small, short-duration, spectacularly un-blinded study.  In the decade that has passed since it was published there seems to have been no further published trials.  On this basis, I must say that I’m deeply unconvinced.

I also think it is clear that the advertising claims made are unsupportable.  This study does not constitute “clinical proof“; neither is there any evidence of “extensive clinical testing“.  Claims that this is a “breakthrough” seem highly premature.

None of this means that this system doesn’t or can’t work.  It means that no one knows.  Until the results of sizable trials that are blind to the ‘rudolph effect’ are published, commercial organizations should not be making these wild claims.

(See this blog post for excellent coverage of this topic from last year)