A canna’ change the laws of physics

Scotty, The Naked Time, stardate 1704.3, Episode 7

BBC editor concedes ‘happiest place’ story is baseless (updated)

Posted by apgaylard on September 5, 2008

[BPSDBThis is just a short post to highlight an interesting discussion that I’ve got into with the BBC’s Health Editor for their News website.  The BBC is carrying an entirely spurious story about the geographical distribution of ‘happiness’ in the UK; both on TV and on-line.

The tell-tale phrase in the on-line article is that, “the researchers stress that the variations between different places in Britain are not statistically significant.”  This appears part way through the piece and entirely scuppers it as a story.  A few ‘bad science’ types have spotted this – notably gimpy – and let the BBC know what they think.

Here’s my first (unusually grumpy) complaint :

“This story is pure bunk.  If, “the researchers stress that the variations between different places in Britain are not statistically significant” then there is no ‘happiest place’ – all places are indistinguishable.  There are no key “factors”.  I can’t believe you carried this non-story on the web site and television.

The real story here is that a study found no significant differences in happiness across the UK regardless of the factors studied.

Even if the differences had reached statistical significance the researchers would still need to show some sort of practical significance.

Even for the BBC this is remarkably statistically illiterate work.  I guess you were just taken in by a shiny press release from some publicity-seeking ‘researchers’.

The Health Editor came back with a prompt, slightly apologetic reply.  Unfortunately this showed that he still didn’t get the point:

Many thanks for your message, and interest in the site.

I take your point. We felt that as the story was of a light-hearted nature, and that as the conclusions were not of great importance, or significance, it made for a light and entertaining read.

The researcher did uncover geographical differences, albeit small.

We did stress in the copy that the differences were not statistically significant, and that factors other than geography were more important.

Kind regards,

Gimpy has explored the ‘entertainment’ angle; so I thought that I’d persist with trying to get the idea across that the study didn’t provide any evidence for real differences.

Thanks for your prompt reply.  I do appreciate a, “light and entertaining read” myself from time to time.  However, when it’s provided by the BBC I expect it to be true.   When you say, “The researcher did uncover geographical differences, albeit small” you are still missing the point.  Failure to reach statistical significance is just a technical way of saying that it is extremely likely (with a probability of usually 19 times in 20, by convention) that the differences are nothing more than the play of random chance.  In other words this study explicitly did not uncover genuine geographical differences.  

Now, it is possible to have a small but statistically significant result.  If this had been the case then your observation would have been correct.  In this case the question to be asked would have been: do the differences matter in any practical way?  However, this study did not even get that far.

It is true that you, “did stress in the copy that the differences were not statistically significant”, however the BBC’s television coverage did not (http://news.bbc.co.uk/1/hi/england/7586919.stm).  

It is interesting that you say, “factors other than geography were more important” did these reach statistical significance?  If they did, why didn’t the coverage focus on them – that would have been a real story.  If they did not, then this is as irrelevant as your main focus on location.   I would really appreciate it if you could clarify the BBC’s position on “light-hearted” news items: am I to consider them as fiction?  This piece was nothing more than fiction masquerading as news.  It serves both the BBC and its license payers badly if the boundary between news and fiction is to be blurred, as it seems to have been in this case. 

It seems that the message has got through, to some extent, as I’ve just received this reply:

Many thanks for your message. Your points are very valid. All I can say is that they will certainly be kept in mind for the future.

Kind regards,

This is all well and good, but the offending items are still available on-line.  My next task is to see if I can get them removed or identified as fiction.

In the grand scheme of things this is small beer.  And I certainly don’t mind the odd light-hearted news story.  I just think that they should be fact not fiction; particularly if public money (from license fee payers like myself) is to be spent, the authority of the BBC is to be invoked and other stories displaced from the news.

EDIT [06-09-2008]

The dialogue continues. After the editor conceded the validity of the criticisms I asked, “Thanks for your prompt and honest reply. When can I expect that the on-line content will be removed or edited to prevent the public from being misled?”

This, for me, is the crunch moment. Will be BBC admit its error and remove the story, or will it leave it on its ‘authoritative’ web-site to continue to misinform the unwary? Generally the BBC seems loath to make corrections or remove stories (though I eventually got the ludicrous piece on LWS ‘treatment’ of SAD removed).

The editor came back with a more defensive reply, “As I have mentioned before, the online story makes explicit reference to the fact that the data was not statistically significant.” I countered with, “Thanks for the prompt reply. You are right – but that’s tantamount to saying that the story has no truth. Looks like I’ll have to ask the ECU to adjudicate on this one.” Raising the spectre of the Editoring Complaints Unit (ECU) seems to have shaken loose a potential concession, “I will beef up the reference regarding the statistical power of the story when I am next in the office on Monday.”

It will be interesting to see what changes in the piece. I’ll still be writing to the ECU, the silly Hay-on-Wye ‘news’ video could do with removing.

EDIT [06-08-2008]

The BBC have edited the story, in an attempt to, “beef up” the statistics. The on-line text has been changed with the addition of a new caveat to the previous nod to statistical significance, “The variations they uncovered between different places in Britain were not statistically significant – suggesting that other studies would come up with different findings.”

Whilst undoubtably true it still doesn’t address the key point: the study found no significant variation by place

As note 7 of the Royal Geographical Society press release points out, “The estimated ranks that were used to produce this press release provide a guide of the order of the differences by area before and after accounting for socio-demographic composition and do not directly imply better or worse places to live in terms of this particular happiness measure. Note that the residuals underlying the ranks for the model that takes into account socio-demographic characteristics are not statistically significant from each other. The only areas that have statistically lower (worse) levels of well-being than the rest of Britain are “Cynon Valley; Rhondda” and “Merthyr Tydfil; Rhymney Valley; Taff-Ely” in Wales. However, these differences are explained by socio-demographic differences in these areas. (emphasis mine)

So, as the Technical Note helpfully puts it, “We find it that socio demographic variations from district to district do explain the differences, in the sense that all places are essentially similar with respect to happiness when we take into account the differences their socio-demographic composition.”(emphasis mine)

15 Responses to “BBC editor concedes ‘happiest place’ story is baseless (updated)”

  1. gimpy said

    Good stuff, more of my correspondence can be found here.

    I’ve been thinking about the BBC’s coverage of science for a while and their frequent misuse of scientific terms such as confusing gene with allele on many of their genetics stories. As I’m sure you appreciate communicating science accurately requires the precise use of technical words as their meaning is defined, understood and largely invariable within the scientific community. I can’t help but wonder if the frequent misuse of such terms by the media when communicating to a lay audience helps fuel the Humpty-Dumpty approach to scientific language of the CAM community, such as Dana Ullman’s recent ludicrous statements about nano meaning powerful.

  2. jdc325 said

    Nice post. I wish you luck with your next task.

  3. apgaylard said

    gimpy: Thanks. I think you’re probably on to something with your point on the lack of precision in the use of language. Once it’s permissible to obfuscate then the door is opened to the Ullman’s of this world, Its what they trade on. (I’m still waiting for him to get back to me after our last exchange…)

    jdc325. Thanks. I’ll need it. The Health Editor has just replied to my suggestion that he should take the story down with, “As I have mentioned before, the online story makes explicit reference to the fact that the data was not statistically significant”. Looks like we’re off to the ECU!

  4. dvnutrix said

    Good letter and yes – unless the BBC is going to introduce an icon of one figure gossiping over a garden wall to another figure to accompany these stories and signify that the same amount of credence should be applied – then they need to be more exact about fact v. fiction.

    frequent misuse of scientific terms such as confusing gene with allele on many of their genetics stories.

    I always find this tricky – yes, absolutely, the terminology must be correct. However, in a lot of technical areas, even when terminology is used correctly, there is no reason that most audiences would appreciate this – or that it educates them to distinguish those times when people (other media) are using terminology in an inappropriate fashion.

    But – this is precisely why online news etc. was supposed to be so transformative, it would allow writers to provide links to background that would fill in all of this additional material for interested parties. Instead – it just created a perception that the BBC and others scramble to put up press releases that they have scarcely fact-checked or ‘commonsense’ checked. On occasion, it has contributed to blurring the fact-fiction line rather than helped to clarify matters.

  5. […] Comments It’s official:… on The BBC mislead, for a la…BBC editor concedes … on The BBC mislead, for a la…Looking Out To Sea … on The BBC mislead, for […]

  6. pv said

    I agree particularly with Dvnutrix comments. Unfortunately the tendency to be imprecise isn’t confined to the BBC or their use of scientific terminology.

    Language fashions and fads arise by and large because of imprecision and misappropriation of words. The current irritating vogue for using the word “multiple” (which has a fairly strict definition) whenever we should say “several”, “many”, “more than one”, “lots”, “different”, or use the prefix “multi-“, is a good example. Whatever the reasons (pretentiousness, laziness, ignorance…) it’s pretty much impossible to stop it.

  7. ambrielle said

    The part of the reply from the BBC that got me was:
    The researcher did uncover geographical differences, albeit small.
    Especially since you had just finished explaining to him that there were no differences.

    Also, I agree with pv; none of this is confined to the BBC. The difference is, you guys are paying for it (not me now thank FSM). One would think you have a right to something other than fiction presented as fact.

  8. apgaylard said

    dvnutrix: I completely agree. The BBC seem to be caught up in churnalism. (Here’s a good BBC editors blog claiming that the big problem is with the newspapers. Oh the irony!)

    pv: couldn’t agree more – though I’m sure that I’m a sinner in this regard to.

    ambrielle: That was my favorite bit as well. I guess my main gripe is not that stories this silly get into the press, but that the BBC is funded out of the pocket of license fee payers – which includes me.

  9. […] It looks like this was originally flagged up by ‘Ithika’, posting in the Bad Science forum, and it has also been blogged about at dougalstanton.net and APGaylard. […]

  10. gimpy said

    Good luck with pressing this issue. Maybe it’s worth pointing out to them that a national newspaper column also covered this?

  11. apgaylard said

    gimpy: Thanks, I’ll certainly do that.

  12. jdc325 said

    I thought this was an interesting section in the BBC blog post you linked to:
    “At the BBC College of Journalism, we place the ethics and values of the trade, along with safeguarding the trust of our audiences, far above any technical or editorial skill… one reason why trust in broadcasting remains much higher than that in the press.”

    Personally, I think removing an untrue story from their website would be very ethical. It would certainly be a step in the right direction in terms of how much I, as a member of their audience, trust them.

  13. apgaylard said

    jdc325. Yes, I saw that and thought that it came over as a wee bit condesending to the press. Given some of the things we have explored on our blogs I think that I would say it’s a great aspiration but the BBC has some way to go. Mistakes are inevitable, but the generally defensive nature of the responses to my criticisms made by editors and the reticence to put things right does not fit in with these ethics.

  14. soveda said

    I still can’t get over the tripe that gets classed in “health”. This sort of non-story should be in their “also in the news” round up along with the skateboarding dogs.

  15. apgaylard said

    soveda: I take your point, it baffles me as well. Thanks for stopping by.

Sorry, the comment form is closed at this time.

%d bloggers like this: