Operation Rudolph
Posted by apgaylard on June 20, 2008
I’ve been talking to Trading Standards and Lloyds Pharmacy about the claims that the latter have been making for their “Allergy Reliever”. This is a ‘medical device’ that allegedly uses phototherapy to relieve the symptoms of allergic rhinitis. Put simply: it shines a red light up your nose.
The claim “Clinically proven” appeared on their website and still appears on the product packaging (see image). Scratching the surface revealed that this claim is based on a single, un-replicated, small, decade-old and deeply flawed trial.
On that basis I am contesting the claim to be “Clinically proven”; Trading Standards have referred the issue to the MHRA and I’m waiting to see what the outcome will be.
In the meantime Lloyds Pharmacy very kindly offered me an “Allergy Reliever” for free. I accepted on the basis that they understood that it does not constitute an endorsement of the product; and that it will not hold me back from making critical observations.
So, Operation Rudolph was born: a home experiment aiming to illustrate what I consider to be one of the major flaws in the trial that underpins the claims made for the product: the design of the placebo, or “sham” treatment.
In my discussions with Trading Standards one of the most difficult things to get across has been my contention that the trial was, in effect, un-blinded by a poorly chosen placebo. So I hope that the pictures in this piece make the point more clearly.
One of the most important features of a placebo is that it is, “indistinguishable from the therapy being evaluated.” Otherwise the patient will know that they are not being treated and are unlikely to experience the benefits of a placebo response. They may even drop out of the trial; they may not see any value in ‘wasting their time’ not getting a therapy for a condition that may be very troubling. These influences will tend to bias the trail in favour of the therapy being tested – making it look more effective than it really is.
Another consequence of a placebo that is not indistinguishable from the therapy being tested is that the people delivering the therapy may be able to see who is being ‘treated’ and who is not. This may introduce biases into their dealings with the trial subjects and their interpretation of the results. Again, this will tend to favour the therapy under test.
A good placebo enables a trial to be designed so that it is ‘blind’, from both the researcher’s and subject’s perspective: the subjects don’t know whether they are getting the therapy or the placebo and neither do the researchers. This is the ‘double-blind‘ methodology that lies at the heart of all fair trials.
So, what about the clinical trial that Lloyds Pharmacy, and others, are relying on? The sham, or placebo, used in the trial was a similar device with its light emitting diodes “internally disconnected“. This means that anyone in the control group was ‘treated’ with a device that emitted no light.
How indistiguishable is this from the treatment? Just have a look at the results from my home experiment (below) The noses in the picture belong to me (I’m sporting my usual post-holiday beard!) and a glamorous female assistant. Now it may be that other people’s noses don’t exhibit the Rudolph Effect to the same degree (age, skin pigmentation and natural variation in the thickness of the structures of the nose could all have a bearing), but this shows that a “no light” placebo is a waste of time. Our red noses are clearly visible to anyone nearby. Importantly, both of us could see our own noses glow red.
So it is clear that when you use a device like this your nose lights up like the proverbial Reindeer; you can see it and so can anyone else nearby. The “no light” placebo is very easily distinguished from the treatment.
Anyone participating in this trial would have known whether they were being treated or not; anyone observing them would also know. It is not clear from the paper whether the participants were ‘treated’ at home, away from the researchers, or in a clinical setting. In the latter case the chances of the researchers knowing who was being treated and who was not is increased.
If the participants were administering the ‘treatment’ themselves then just switching on the device prior to insertion, out of curiosity (and who wouldn’t) would reveal what part of the study they belonged to: treatment or control (See the last image above).
If the device tested in the clinical trail was similar to Lloyds Pharmacy’s product, and one should hope that it is given that this trial is their sole evidence of efficacy, then light leakage through the nose and along the probes will have the same effect.
That the authors did not go to the trouble of developing a proper placebo raises concerns about the general quality of the trail. Accepting what is in a published paper requires a degree of trust; when the work is this sloppy that is undermined.
Does this mean that this device doesn’t work? No, but in the absence of good quality trials all claims for this product should be treated with extreme scepticism.
Conflict of Interest
As disclosed in this piece I am now in the pay of ‘big quacka’, having received an alternative medical device worth £13.49 for free. Having sold my soul for such a princely sum, any future endorsements that I may make of CAM modalities should be seen in this context.
27 Responses to “Operation Rudolph”
Sorry, the comment form is closed at this time.
A canna’ change the laws of physics 
dvnutrix said
If you were wearing wrap-around shades and a balaclava, and you and those around you undertook to keep your eyes closed, a blind man on a horse wouldn’t notice, as my mother would have said.
I assume not, but was there any sensation of warmth?
I take it that you shall be printing out this post, complete with photos on nice glossy paper, and bundling it off to both Lloyds and Trading Standards. If you can find it in your heart, perhaps Dr Chris Steele would also benefit. After all, Steele endorses the Medinose Plus version while saying he doesn’t: I’ve checked and the promo is still there.
claireob said
I’m sure this would provide quality entertainment for neighbourhood children – and probably stop the poor unfortunate with the ‘very shiny nose’ complaining about hayfever symptoms.
But seriously, I was in the Lloyds Pharmacy branch which is in the same building complex as our GP practice yesterday, to pick up a prescription. Displayed prominently were Medinose, air ionisers and other expensive kit curiously not supported by guidelines such as BTS/SIGN (asthma) or ARIA (rhinitis). I have mentioned this lack of evidence previously to staff & got the reply ‘some people find it helpful’.
raygirvan said
Oops – sorry. Should have RTFM re HTML tags. Just wanted to link here.
apgaylard said
Thanks for the comments.
dvnutrix:
No sensation of warmth beyond what you’d expect sticking a tube up your nose. LED’s are very efficient, so don’t dissipate much heat. A quick hand-calc suggests that the incident illumination shouldn’t be much of a give-away either.
I’ll get in touch with Dr Steele and see what he has to say. As for using this piece with the various agencies, I think that the MHRA may be the people I’ll try these technical arguments on. Trading Standards have been very helpful but they admit that these sorts of arguments are a bit beyond them.
claireob:
Couldn’t agree more. You can add magnets, copper bracelets and homeopathy to the list for pharmacies in my area. David Colquhoun has been blogging very effectively on this issue for some time, with a particular focus on Boots.
The more I look at what they are prepared to sell the more I worry about them (strictly the pharmacists that work for them) taking a more prominent role in the delivery of healthcare in the UK the more concerned I get.
Of course, to be fair, pharmacies in other countries are just as bad: the French ones I have seen seem obsessed with homeopathy.
raygirvan:
Cool. Thanks.
draust said
AP – you rule.
Seriously, a wonderful post exposing the fallacies of non-indistinguishable placebos. The photos are priceless.
Next time I talk to the med students about placebos and blinding I will be starting here. Will get the basic concept across beautifully before we go on to talk about the things like observer bias and “active placebos” for drugs with easily recognisable unwanted effects, like omega-3s (fishy burps) and SSRIs (take your pick of a list).
Anyway, great stuff.
claireob said
It seems you can also access these tests through LLoyds:
http://www.lloydspharmacy.com/wps/portal/products/medicines/allergy/foodallergies
“Lloydspharmacy Food Intolerance Test
Working in partnership with YORKTEST Laboratories, we offer a unique and revolutionary food intolerance YES/NO test.
Designed for people suffering from chronic conditions or less than satisfactory health, the test analyses a tiny sample of blood to see if food intolerance exists, and therefore whether or not your symptoms could be food related.
If your results are positive, the second stage of the test is available which further analyses your blood sample for intolerances against 113 different foods.
Please note this test is only available in selected pharmacies.”
No mention of the dim view taken by the House of Lords Allergy Report or this from the BSACI’s reponse to same:
“… * Allergy self-testing kits should be interpreted with advice from trained healthcare personnel and IgG food antibody tests should not be used to diagnose food intolerance in the absence of stringent scientific evidence.”
apgaylard said
draust:
Thanks. Please feel free to make educational use of the photos.
apgaylard said
claireob:
That’s sad; eloquent testimony to where commercial conflicts of interest can get you in healthcare.
claireob said
Just to add that ‘clinically proven’ was prominently visible on the Medinose on sale in Lloyds yesterday.
Commercial companies appear eager to take advantage of developments in healthcare provision, e.g.
http://www.responsesource.com/releases/rel_display.php?relid=38912&hilite=
“…GP and medical advisor, Dr Hilary Jones says:
“Allergies are dramatically on the increase yet reliable professional help remains elusive. Training the UK’s 31,000 pharmacists to assist sufferers in identifying, avoiding and treating an allergic reaction is a practical and logical step, especially as pharmacists are ideally placed amongst other things to stock, prescribe and dispense finger prick tests such as those from YorkTest…”
mjsimpson said
To be fair – and we must be fair – trial participants using the placebo nose-thing would not know that the red light was visible through the skin of the nose unless they had seen someone using a real nose-thing. Most people would assume that the skin and flesh on their schnozz is not translucent. So if all the participants in the trial were isolated from each other, those using the real nose-thing would know it was real but those using the placebo nose-thing would also assume it was real because they would have no reason to expect the Rudolph effect. Unless they were in the habit of shinning lights up their nose.
Admittedly just turning the device on when not nasally inserted would very obviously indicate whether it was real or placebo but this is the important thing here. The rudolph effect, while very amusing and photogenic, is irrelevant unless the subjects are on contact with each other.
apgaylard said
mjsimpson:
Thanks for your comment. I do agree that we need to be fair. Perhaps I was an oddly inquistive child, but I did find out quite early that light could shine through the nose (I’ll spare everyone the elaboration).
I take your point that the trial setting is important – it’s not described in the paper. However, I think that there’s no way around those in the treatment group knowing they were being treated (via the Rudolph Effect)- that breaks blinding. As for the control group it depends, as you point out, on their expectations. This in turn depends, in part, on what the trial organisers told them – again this is something we don’t know.
The other thing that strikes me is that just looking at the device it’s clear that it had LEDs in the probes. To me, that would raise an expectation of illumination. But, perhaps I’m atypically curious (plus the ability to identify such devices is also a function of my education and early professional experiences).
It’s also notable that the more recent work done on phototherapy for allergic rhinitis has used a sham illumination (weak white light), though perhaps they were helped by their prefered part of the spectrum being UV/VIS. This may indicate a more careful approach to sham design is emerging in this field.
claireob said
The AAAAI has commented on the 2005 JACI paper here:
http://www.aaaai.org/AADMC/CURRENTLITERATURE/selectedarticles/2005archive/rhinophototherapy.html
Also, referring to that paper, the 2007 ARIA guideline on Allergic Rhinitis (and its impact on asthma) states that more data are needed, using simpler equipment.
apgaylard said
claireob:
Thanks. That’s interesting.
casperg said
Agreed the trial setting would be important — did the subjects even know they were participating in a phototherapy trial? If not, an absence of light might not have been a problem. After all, the device could have been releasing ultrasound/magnetism/positive energies etc, any one of which would probably be really beneficial. Or not.
Seriously, if I’d been in the trial and had noticed my nose lighting up I’d probably have concluded it was a pretty elaborate sham therapy and decided I was in the placebo group. Then again, maybe it is an elaborate sham therapy, and the placebo group consists of the paying customers. There goes the blinding….
apgaylard said
casperg:
A fair comment. I’ve submitted a comment to the Editors of the journal that published the trial (Ann. Allergy Asthma Immunol.); maybe this might get some comment from the authors – though after 10 years they may not be active.
Top Posts « WordPress.com said
[…] Operation Rudolph [image]I’ve been talking to Trading Standards and Lloyds Pharmacy about the claims that the latter have been […] […]
thestopgap said
I am a chronic hay fever sufferer; I am lucky that Benadryll spot relief type tablets completely eliminate my symptoms, many are not. I get all the usual symptoms from itchy eyes and runny nose, to feeling the need to tear the lining of my mouth out just to stop the itching, it’s just horrendous when it hits. I have before now even experienced bleeding in the corners of my eyes from the irritation – oh happy day!
My father saw and bought for me the Lloyds version of this, the Medinose phototherapy device. I was somewhat sceptical to say the least of it but decided to try it anyway. The results are quite astonishing and repeatable and I find it extremely difficult to attribute them to a placebo effect. Without fail it results in near total reduction in itching, no further desire to sneeze, clearing of mucus and sinus pressure (I won’t get too gory on the details) and most noticeably, a massive and rapid reduction in the itchy roof of the mouth symptoms (which even on Benadryll takes hours to subside to this level). I have had extensive hypnotherapy to lessen the hay fever symptoms (amongst other things including curing sea sickness, something I had always suffered from and medications could not prevent). It had a significant effect at the time but since moving to the country where the pollen is worse, so too was the severity of the hay fever. I know that suggestion works on me with this specific condition and so I could never rule out placebo effect; I have a very strong and grounded faith in hypnotherapy but did not once achieve results even close to the efficacy or speediness of this device.
Having studied experimental design (including double blind experiments and so on) in psychology I am aware of the overriding need to prevent bias in the data with careful design. Often this is achieved by informing both subjects and testers that they are undertaking a completely different experiment and certainly never giving them any expectations of what apparatus or testing materials would do (unless this was ethically necessary or certain technical knowledge was required on the subjects’ part to enable participation).
The paper in question may not have been as thorough as it should have been (any paper worth its salt should enable a lay person to understand the experiment and in brief the conclusions, and a scientist to repeat the experiment if not the results). This does not however mean that the experiment was flawed, there are many ways in which the subjects could have been shielded from the glowing nose issues. For example they could have been told that the light was merely to indicate that the device was on, not that it was the method of treatment under examination. Placebo subjects could have been informed that the device emitted only invisible wavelength light and hence would see no visible sign of operation (and back this up with audible operation notification to ensure fair results with candidates told that the light meant operation of the device and nothing more). LEDs are of course capable of emitting infra red light which is invisible to the human eye so even the scientifically curious would have a perfectly believable explanation.
The most stable and suitable test however would seem to be isolated test subjects who were given no information as to how the device worked or when it was working, possibly even blindfolded or blinded with bright lights to prevent visual cues. Is there actually anything to suggest that these measures were not undertaken, even if they are not explicitly detailed in the report? I’d be interested to see a copy of the published paper if you could point me in the right direction.
dvnutrix said
From the link in the post: Neuman and Finkelstein (pdf). You will see that the “sham illumination” involved no illumination.
Whatever the means, may I say how pleased I am to learn that your hayfever symptoms have improved (the eye-blood part sounds more like mediaeval torture).
apgaylard said
Thestopgap,
Thanks for an interesting comment. You’ll see that most of the questions you pose about the trial setting cannot be answered from the paper. (In case you didn’t spot it I disussed the paper more generally here.)
Note, I’m not saying that this can’t work; rather that the evidence base is insufficient to support a general claim to be “Clinically proven”.
One of the other commenters also raised the issue of “active placebo”. I wonder whether the spectacular nasal illumination experienced by the treatment group could also be seen in those terms? (I’d be grateful for any insights on this point)
By the way, my wife is giving it a go: no improvement as yet (some deterioration if anything) though this doesn’t really provide any insight (I’m glad your symptoms are improved though, it’s a miserable ailment).
jdc325 said
Shades of SciencePunk’s detox pads challenge. I think perhaps this is what we need more of at Bad Science Blogs Dot Net… home experiments! You certainly picked a good one to illustrate AP – the Rudolph Effect is really quite striking in the slides you added.
When you look at the obvious failure of blinding in this trial alongside the small number of subjects in the various groups (one in the polyposis control group), not to mention the failure to adjust the significance level to take account of the number of variables being measured, I think the only conclusion you could reach is that this study cannot possibly provide clinical proof that the LEDs are effective. I’ll be interested to see what the MHRA have to say on this.
apgaylard said
jdc325:
Thanks for the comment. I’d like to do a bit more experimentation. This is the first chance that I’ve identified. I have a couple of other ideas related to my professional interests.
I’m surpised my better half consented to be photographed for all to see, when I’d interupted her cleaning after we had come back froma short break and wearing no make-up!
As you say the polyposis element of this work is very thin. I wondered why they bothered to publish it at all? On the rhinitis trial I’d agree with you entirely. The best that could be said on this work is ‘more research’ not it’s ‘clinically proven.’
furbearingbrick said
Uh-oh, Thestopgap has committed Doggerel #14 (“It Works THROUGH the Placebo Effect!”)
furbearingbrick said
Grrr, WordPress ate my blockquote. Here it is, from Bronze Dog’s blog:
[One of the funniest arguments I’ve heard from woos is that their magic pills work through the placebo effect, and I don’t think they mean it in a Zen “acting by not acting” sort of way.
The placebo effect is what you get when you do nothing, but think you’re doing something: It’s a combination of two things: Psychological tricks and coincidental recovery.
First, are the psychological tricks: Confirmation bias, the regressive fallacy, misattribution, subjective validation, and so on and so forth: We essentially fool ourselves into thinking the symptoms aren’t as bad, or not as common, etcetera. They’re still there, but we have an excuse to ignore them: They don’t fit with expectations, so we rationalize them away. There’s no “power of positive thinking” going on, just a conscious and unconscious motivation to ignore what doesn’t fit. This is why double-blinding is important: It make most of those rationalizations very implausible, and points of view irrelevant: Belief isn’t going to change the numbers.
Second, there is the physical aspect: Natural improvement and coincidental recovery. It could be that a person’s immune system is rallying for a final assault when he takes the treatment. There’s no way to be certain that it’s the drug that’s doing anything. That’s where the placebo control in studies comes in: Compare the drug to essentially doing nothing as a base line. If there’s a significant difference between the placebo group and the treatment group, it’s unlikely to be coincidence. The bigger the study, the less likely coincidences become.
There are probably more aspects to placebos and nocebos, but that should be the bulk of it: The placebo effect is what happens when you merely think you’re doing something.]
apgaylard said
furbearingbrick: Thanks for the summary. I’d also suggest conditioning. R Barker Bausell’s excellent Snake Oil Science has some very interesting observations on this. There seems to be pretty reasonable evidence that placebos in both people and animals can elcit biochemical changes via this mechanism; which is essentially that shown by Pavlov with his salivating dogs. This is interesting because it is one of the main reasons that advocates of homeopathy get it wrong when they cite animal studies as being immune to the placebo effect. (the other obvious one being that often these interventions rely on the subjective impression of people who know what is going on to assess how the animals are doing.)
The best way to cure hayfever is to shove a light up your nose | Plausibility.net said
[…] And more Home | Top | Design | Plausibility is: © Copyright 2008 Plausibility.net. var gaJsHost = ((“https:” == document.location.protocol) ? “https://ssl.” : “http://www.”); document.write(unescape(“%3Cscript src='” + gaJsHost + “google-analytics.com/ga.js’ type=’text/javascript’%3E%3C/script%3E”)); var pageTracker = _gat._getTracker(“UA-5027707-15”); pageTracker._initData(); pageTracker._trackPageview(); […]
salerio said
Hi
I know i’m a couple of years late, but I thought you might be interested in this that I posted to the JREF forum. I hadn’t come across your BLOG until I had a reply and thought I would give you the heads-up. I could have saved myself work just by copying and pasting this BLOG to the ASA rather than spending hours on Springer and PubMed. I actually headed my post “Looking like Rudolf” oh well great minds and all that.
Interesting BLOG btw.
wordpress@etiqa.co.uk
“I’ll give ’em red lights to stick up your nose!
I was a bit surprised that I was the only one who complained about this. As a hay-fever sufferer I got fed up with these flaming adverts, I did a full literature search on the subject of light therapy and allergic rhinitis, read more papers about snotty noses than I want to read in my entire lifetime and put a case together for the ASA complaining about the advert.
Read all about it
http://www.asa.org.uk/asa/adjudicati…_ADJ_47695.htm
Typical example http://www.youtube.com/watch?v=B0Gu5y2MCBY it got so bad it was like spamming for v!agr4, every advert break, which made me more ticked off while i was stuck inside all summer snuffling to myself”
apgaylard said
Thanks for the kind comment. Well done for sticking with your complaint.