This comes up now and again, and the world of polling comes in for a lot of stick, but since the election I have noticed a lot of attacks on YouGov particularly from Liberal Democrat activists. The argument is that YouGov is manipulating their polls to show worse results for the Liberal Democrats. Certainly the Lib Dems have shown repeatedly lower numbers on YouGov polls but does this claim really stand up to methodological analysis?
The first target for criticism is usually YouGov’s method of party identification weighting. All pollsters attempt to match the political views of respondents to the political views of the electorate at large. Most do this by weighting to a close approximation of the last election result. YouGov instead weight by party identification, that is to say how someone responds if you ask the question ‘Which party do you support’. This means YouGov polls on the assumption that about 12% of people claim to be Liberal Democrats, however political scientist Antony Wells, who works with YouGov, notes that YouGov polls actually have a higher number of people who voted Liberal Democrat at the last election in their samples than other pollsters. Clearly they are not weighting against the party then, or if they are, they are quite bad at it.
After the 2010 election Wells freely admits there were minor changes to the way YouGov does it polling. Yet it should be remembered that before the election YouGov were showing very high results for the Lib Dems, and continued to until Election Day. The fact that YouGov was fairly well out on the Lib Dems justifies some changes, and YouGov’s changes appear effective. When YouGov test polled and asked people how they voted at the last election with their new methodology results were generally accurate to within 1%.
So why does YouGov show worse results for the Lib Dems than other pollsters? The main reason is how pollsters have responded to the ‘Shy Tory’ effect. This was an effect noted in the 1992 election where the polls showed Labour ahead of the Conservatives and were then proved completely wrong on election night where John Major’s Tories ended up defeating Labour by 7.5% of the vote. The reason for this, it was decided, was the ‘Shy Tory’ vote; Conservative-leaning voters who responded ‘Don’t Know’ to pollsters because being a Tory was not considered a particularly positive attribute by that point. Different pollsters responded in different ways. ComRes, for example, asks ‘Don’t Knows’ who they’ll vote for again, but more forcefully, whereas ICM assumes that 50% of Don’t Knows will vote the same as at the prior election. In a sense such methodologies can be seen as questionable – after all you are then not reporting what people have said, but pollsters would argue that such methodologies are justified by the ends. ICM’s method, which is by far the one which tinkers the most, is also the one which produced the most accurate result in 2010, joint most accurate with Populus, who use a similar methodology. These two pollsters have, since the last election, shown the highest results for the Lib Dems. This may seem like good news for the Lib Dems, but things have changed now the Lib Dems are in government. What we do know however is that there are a lot of former Lib Dems in the polls who are now ‘Don’t Knows’ and who are being redistributed by ICM and Populus back in the Lib Dems general direction. We may therefore be seeing a ‘Shy Lib Dem’ effect in the polls, certainly being a Lib Dem is not perceived as being as cool as it was before the election.
YouGov does not do this, because it is a very different type of pollster. ICM, Populus et al poll use telephone methodology, on the most part. Part of the ‘Shy Tory’ theory is the belief that telling a person that you support an unfashionable political party is slightly embarrassing. YouGov, however, polls online, so there are no people. Telling a drop down box which party you support is much less shaming. YouGov, therefore, in theory, should not be effected by a ‘Shy Tory’ effect, and this is one of the many perceived advantages of polling online. So while YouGov is different, it is not necessarily wrong.
Similar accusations of YouGov bias have been thrown over AV polling. YouGov is the only pollster which shows ‘No’ ahead of ‘Yes’. This has mostly been placed at the foot of YouGov’s question which asks the following:
“The Conservative-Liberal Democrat Government are committed to holding a referendum on changing the electoral system from first-past-the-post (FPTP) to the Alternative Vote (AV). At the moment, under first-past-the-post (FPTP), voters select ONE candidate, and the candidate with the most votes wins. It has been suggested that this system should be replaced by the Alternative Vote (AV). Voters would RANK a number of candidates from a list. If a candidates wins more than half of the ‘1st’ votes, a winner is declared. If not, the least popular candidates are eliminated from the contest, and their supporters’ subsequent preferences counted and shared accordingly between the remaining candidates. This process continues until an outright winner is declared. If a referendum were held tomorrow on whether to stick with first-past-the-post or switch to the Alternative Vote for electing MPs, how would you vote?”
Lib Dem activists and electoral reformers have argued that this question is unfair because it specifically mentions the AV referendum as coalition policy, suggesting ‘no’ is winning due to coalition policy. However when YouGov asked the question again without mentioning the government they got the exact same result. Rather, it seems, respondents were put off by the long explanation of AV, and why wouldn’t they? FPTP can be explained in a sentence: ‘the guy with the most votes wins’ there, done. That’s a horrible wall of text by comparison (indeed I bet half of you didn’t even read it). There is some validity to this method, however.
A January poll from Angus Reid showed that 54% of respondents said they were either ‘not at all informed’ or ‘not very informed’ about AV. Therefore getting a measure on support by simply asking the ballot question is a bit dodgy, as many people likely have no idea what AV is, or do not properly understand the system. Indeed this is also indicated by the high number of people in all polls saying they are undecided. YouGov’s question therefore, at least attempts to explain what AV is, whereas people may be saying they support it to other pollsters for a variety of other reasons. For example they may think AV is Proportional Representation or they may simply like the idea of changing the system in general.
Critics of YouGov claim that, as a customer of the Sun, it is in hoc to the Murdoch Press. However this is a great misunderstanding of the business model of polling firms. Companies like YouGov and ICM do not get the majority of their business from polling, in fact for companies like this polling represents as little as 2% of business. They are market research companies first and foremost. Polls can actually be a bit of a pain in the neck for such companies, but their benefits are that they raise public awareness and brand recognition. If a pollster gets a bad reputation from bad poll results, therefore, this can have knock-on effects in the rest of their business. Sacrificing a professional reputation is not worth a few extra bob from Rupert Murdoch.
I would also like to produce a warning from recent history. In 2008 three pollsters were polling for the London Mayoral Election, the two others showed the contest as having a slight Ken Livingstone lead. YouGov was the exception, showing a wide lead for Boris. Ken actually issued a complaint against YouGov saying that their methodology was ‘fundamentally flawed’. While the specific poll that caused the complaint was probably a slight rogue, in the end YouGov was the most accurate of the three pollsters that produced polls during the election, and with regards to Ken and Boris was almost exactly on the money.
To finish if anyone from YouGov has been reading this and would like to offer anyone at Britain-Votes a job I’m sure it would be appreciated... *cough*
Whilst methodology ia part of the reason between the LibDem figures given by Yougov and ICM , it is not in fact the main cause . Yougov simply do not find as many Libdem supporters in their samples than the other pollsters BEFORE any weighting adjustments .
ReplyDeleteOver the last 2 weeks the average LibDem figure in Yougov polls before weighting was 10% after weighting the final average figure is 9% .
ICM had 14.3% LibDems in their last poll BEFORE weighting . Initial weightings dropped that to 13% before final allocations of don't knows upped it to 15% . The last Ipsos Mori poll in fact had 15.3% Libdems in their initial sample before weighting adjustments reduced it to 13% .