Newspaper article writers love to take the results and conclusions of polls (especially fundamentally flawed ones) and to perform further extrapolation upon these results in order to jump to further ‘interesting’ conclusions.
It can be educational to examine these things more closely, which is precisely what the typical newspaper reader/commenter doesn’t bother to do, in an entirely predictable example of confirmation bias, which the writer is clearly aware of and hoping for.
The first thing to do when confronted with a poll is to search for the sample size – search being the operative word, because it’s typical to find that those who publish their conclusions based upon these polls tend to make this information difficult to locate.
Secondly, one should look for some kind of bias regarding the demographics of those being polled. So for example, a poll about attitudes to child-rearing might provide polar-opposite conclusions if the sample is made up entirely of mothers as opposed to fathers, or parents as opposed to non-parents.
Here is an interesting example that I stumbled across today –
Headline in newspaper – (from this link)
Lifestyle ‘is better in Poland than Britain’: Less crime and violence – and it’s cheaper too.
It’s important to take a moment to stop and consider the headline (above), because it’s likely to demonstrate the main deception that the writer is attemtping to make.
Just 5 per cent of Britons describe themselves as happy
Poland, France, Spain and Italy all have less material wealth, but fared better than the UK
Next we should examine the sub-heading (above), which will probably attempt to reinforce the deception as well as sometimes also doing the opposite – by offering a little truth in order to water down the deception of the main heading (which everyone will read) the sub-heading can also reassure the sceptical (more likely to also read the sub-heading) via this watering-down process.
The main fallacy used by polls and their resulting write-ups is the unrepresentative sample fallacy which can clearly be observed throughout the headline and sub-heading. Specifically, unless the whole of Britain is polled it is extremely poor and misleading journalism to use a sub-heading such as ‘Just 5 per cent of Britons describe themselves as happy.’
There are many more lessons to be found regarding propaganda within the article (for example, to ask yourself ‘what are they trying to achieve via this deception?’) but let’s move on to the first and most necessary part of our process here – finding the actual poll that the article is written about.
You will notice that nowhere in the actual article does it explain where the poll results are derived from. The only place where I can see it is at the bottom right of the first graphic – ‘Source -USwitch.com’.
So we head off to Google search engine and a quick search for ‘USwitch poll’ returns two pages and one of them is the poll we are looking for – USwitch poll August 2011.
Of course, the first consideration is that this is a poll (apparently – but see below for clarification) conducted by a commercial company (not a dedicated polling company) which is in the business of providing online price comparisons of various consumer utilities and everyday purchases, such as insurance and communications contracts.
Next we should eagerly look for the sample size of people polled and any demographic bias. What we find is that many of the statistics used for the pollsters to arrive at their own conclusions are firstly (1) drawn from a variety of sources and secondly (2) these statistics are manipulated somewhat using formulae devised by the pollsters.
Examples –
1) the poll is designed to compare Britons with the rest of Europe, but some of the statistics used are derived from sources which use international statistics – EG –
Working hours based on average usual weekly hours worked on main job. data from OECD/ ILO http://laborsta.ilo.org
2) an example of the formulae –
Income data is net income after taxes (in GBP) for a two-earner married couple, one at 100% of average earnings and the other at 33% with 2 children. Data from from the Taxing Wages reports by OECD (these reports are currently not available on the OECD website).
So already, we have found two areas where the statistics are being skewed, before the newspaper gets their hands on them and skews them even more with creative headlines.
But back to the core fallacy, the unrepresentative sample. It’s right at the bottom of the article on the USwitch page –
The accompanying poll was carried out by Opinium Research. Sample size of 2,036 Nationally Representative (UK adults aged 18+) between 26th and 29th August 2011.
So the poll was actually conducted by a dedicated polling company. But who in their right mind would suggest that you can accurately define the attitudes of a nation with a population of over 61 million (according to 2006 estimates) by polling 2,036 people?
This is roughly one three-hundredth of a per cent! That’s 0.00333(reoccurring) of a percent.
But there’s more. Now that we have found the company who conducted the poll, let’s just take a brief look at their website.
I found this page called ‘Research based PR stories – fact or fiction?’
Extracts –
It was Benjamin Disraeli who said that, ‘There are lies, damned lies and statistics’. He might have been referring to some of the research-based PR stories that crop up in the media. A headline such as ’80 per cent of Brits face pension misery’ may grab the attention, but, if the research behind the headline is of dubious quality, then so is the story.
For me, the real culprit, however, is poor questionnaire design. Put yourself in the position of a panellist taking part in an omnibus survey. You’ll be expected to answer questions on a wide variety of topics. If you don’t know an answer, chances are you’ll take a guess, as opposed to ticking a ‘don’t know’ box. This is a particular problem in financial research, where most questions revolve around how much people earn, borrow, save, invest and spend. I don’t know about you but, off the top of my head, I couldn’t tell you what interest rate I’m paying currently on my mortgage or what my annual car insurance premium is. And I might not want to admit my ignorance, so would probably take an educated guess. If I’m not alone in doing this, then the survey findings – and resultant statistics – are going to be completely skewed.
Market research is great at producing headline-grabbing news stories, but ‘damned lies’, as Disraeli put it, are not worth the paper they’re written on.
It’s nice to see that the company responsible for this poll feel exactly the same way that I do about propaganda. 😉 No doubt they will be releasing a press-release shortly to explain their decisions regarding their interesting formulae, their ridiculously low sample size and to rebuke the Daily Mail for their use of this poll to construct their umm, ‘conclusions’?
The point which they make in the middle quote (above) is entirely valid. We have no idea what form the questioning in this poll took. But anyone with a brain knows that you can easily skew the results of a poll before it is taken by using multiple-choice instead of open questions, with the choices specifically chosen in advance for nefarious reasons.
When challenged on this, it is easy to defend by stating that it’s too much work for researchers to extrapolate results from the answers to open questions (so intead we use a computer to read the results), plus one could also point out that the interpretation of the answers by the researchers could be used to add bias to the findings – therefore multiple choice is ‘more accurate’.
The whole point of my article is to demonstrate the educational value of paying attention to these polls, the sample size, the demographics (or lack of information about them) and how these results are taken by other sources and extrapolated into sweeping generalisations.
What lessons can we learn about polls, statistics, newspapers and those who read them and take their findings at face-value? For example, as of this time (9am on the morning of publication) there are 190 comments on this article and only one that I can find, where the commenter doesn’t appear to be completely taken in by the deception. He calls himself ‘ExRat’ 😉
I hope that you find this useful and educational.
‘Anyone who doesn’t take truth seriously in small matters cannot be trusted in large ones either.’ Albert Einstein.
‘It’s no wonder that truth is stranger than fiction. Fiction has to make sense.’ Mark Twain
There are only two mistakes one can make along the road to truth; not going all the way, and not starting.’ Buddha.
‘A lie gets halfway around the world before the truth has a chance to get its pants on.’ Winston Churchill.
Updated –
I knew there was another article similar to this which used a sample of 2000 people in a poll to draw sweeping conclusions about the British consumer, although at least this one clearly points out the sample size in the article, right at the end of it –
Consumer Confidence Improves For First Time In Five Months
As you can see, this article is a fair bit more informative and factual than the previous one, but still – why do they choose to survey only 2000 people?
The GfK survey was based on a sample of around 2,000 people and conducted between September 2 and September 11 on behalf of the European Commission
Things that make you go hmmm.