BBC editor concedes ‘happiest place’ story is baseless (updated)
Posted by apgaylard on September 5, 2008
[BPSDB] This is just a short post to highlight an interesting discussion that I’ve got into with the BBC’s Health Editor for their News website. The BBC is carrying an entirely spurious story about the geographical distribution of ‘happiness’ in the UK; both on TV and on-line.
The tell-tale phrase in the on-line article is that, “the researchers stress that the variations between different places in Britain are not statistically significant.” This appears part way through the piece and entirely scuppers it as a story. A few ‘bad science’ types have spotted this – notably gimpy – and let the BBC know what they think.
Here’s my first (unusually grumpy) complaint :
“This story is pure bunk. If, “the researchers stress that the variations between different places in Britain are not statistically significant” then there is no ‘happiest place’ – all places are indistinguishable. There are no key “factors”. I can’t believe you carried this non-story on the web site and television.
The real story here is that a study found no significant differences in happiness across the UK regardless of the factors studied.
Even if the differences had reached statistical significance the researchers would still need to show some sort of practical significance.
Even for the BBC this is remarkably statistically illiterate work. I guess you were just taken in by a shiny press release from some publicity-seeking ‘researchers’.
The Health Editor came back with a prompt, slightly apologetic reply. Unfortunately this showed that he still didn’t get the point:
Many thanks for your message, and interest in the site.
I take your point. We felt that as the story was of a light-hearted nature, and that as the conclusions were not of great importance, or significance, it made for a light and entertaining read.
The researcher did uncover geographical differences, albeit small.
We did stress in the copy that the differences were not statistically significant, and that factors other than geography were more important.
Gimpy has explored the ‘entertainment’ angle; so I thought that I’d persist with trying to get the idea across that the study didn’t provide any evidence for real differences.
Thanks for your prompt reply. I do appreciate a, “light and entertaining read” myself from time to time. However, when it’s provided by the BBC I expect it to be true. When you say, “The researcher did uncover geographical differences, albeit small” you are still missing the point. Failure to reach statistical significance is just a technical way of saying that it is extremely likely (with a probability of usually 19 times in 20, by convention) that the differences are nothing more than the play of random chance. In other words this study explicitly did not uncover genuine geographical differences.
Now, it is possible to have a small but statistically significant result. If this had been the case then your observation would have been correct. In this case the question to be asked would have been: do the differences matter in any practical way? However, this study did not even get that far.
It is true that you, “did stress in the copy that the differences were not statistically significant”, however the BBC’s television coverage did not (http://news.bbc.co.uk/1/hi/england/7586919.stm).
It is interesting that you say, “factors other than geography were more important” did these reach statistical significance? If they did, why didn’t the coverage focus on them – that would have been a real story. If they did not, then this is as irrelevant as your main focus on location. I would really appreciate it if you could clarify the BBC’s position on “light-hearted” news items: am I to consider them as fiction? This piece was nothing more than fiction masquerading as news. It serves both the BBC and its license payers badly if the boundary between news and fiction is to be blurred, as it seems to have been in this case.
It seems that the message has got through, to some extent, as I’ve just received this reply:
Many thanks for your message. Your points are very valid. All I can say is that they will certainly be kept in mind for the future.
This is all well and good, but the offending items are still available on-line. My next task is to see if I can get them removed or identified as fiction.
In the grand scheme of things this is small beer. And I certainly don’t mind the odd light-hearted news story. I just think that they should be fact not fiction; particularly if public money (from license fee payers like myself) is to be spent, the authority of the BBC is to be invoked and other stories displaced from the news.
The dialogue continues. After the editor conceded the validity of the criticisms I asked, “Thanks for your prompt and honest reply. When can I expect that the on-line content will be removed or edited to prevent the public from being misled?”
This, for me, is the crunch moment. Will be BBC admit its error and remove the story, or will it leave it on its ‘authoritative’ web-site to continue to misinform the unwary? Generally the BBC seems loath to make corrections or remove stories (though I eventually got the ludicrous piece on LWS ‘treatment’ of SAD removed).
The editor came back with a more defensive reply, “As I have mentioned before, the online story makes explicit reference to the fact that the data was not statistically significant.” I countered with, “Thanks for the prompt reply. You are right – but that’s tantamount to saying that the story has no truth. Looks like I’ll have to ask the ECU to adjudicate on this one.” Raising the spectre of the Editoring Complaints Unit (ECU) seems to have shaken loose a potential concession, “I will beef up the reference regarding the statistical power of the story when I am next in the office on Monday.”
It will be interesting to see what changes in the piece. I’ll still be writing to the ECU, the silly Hay-on-Wye ‘news’ video could do with removing.
The BBC have edited the story, in an attempt to, “beef up” the statistics. The on-line text has been changed with the addition of a new caveat to the previous nod to statistical significance, “The variations they uncovered between different places in Britain were not statistically significant – suggesting that other studies would come up with different findings.”
Whilst undoubtably true it still doesn’t address the key point: the study found no significant variation by place
As note 7 of the Royal Geographical Society press release points out, “The estimated ranks that were used to produce this press release provide a guide of the order of the differences by area before and after accounting for socio-demographic composition and do not directly imply better or worse places to live in terms of this particular happiness measure. Note that the residuals underlying the ranks for the model that takes into account socio-demographic characteristics are not statistically significant from each other. The only areas that have statistically lower (worse) levels of well-being than the rest of Britain are “Cynon Valley; Rhondda” and “Merthyr Tydfil; Rhymney Valley; Taff-Ely” in Wales. However, these differences are explained by socio-demographic differences in these areas. (emphasis mine)
So, as the Technical Note helpfully puts it, “We find it that socio demographic variations from district to district do explain the differences, in the sense that all places are essentially similar with respect to happiness when we take into account the differences their socio-demographic composition.”(emphasis mine)
15 Responses to “BBC editor concedes ‘happiest place’ story is baseless (updated)”
Sorry, the comment form is closed at this time.