Evaluating the Polls: an Open Letter to Ontario’s Journalists
Wednesday, September 14, 2011
Toronto, ON – We’ve all seen over the last few days a confusing cacophony of polls on the Ontario election. Depending on what poll you read, McGuinty's Liberals are on a roll, Hudak and the Tories are comfortably ahead, or the Grits and Tories are neck and neck. How can this be? It is because all polls are NOT created equally. And, in spite of what you may assume, pollsters are never held to account for their indiscretions, incompetence and mistakes (there is no “polling police”). Some marginal pollsters count on your ignorance and hunger to make the news to peddle an inferior product. Others are using your coverage to "prove" that their untried methodology is the way forward for market research in Canada. Instead of being their own biggest sceptics (which is what our training tells us to be), they've become hucksters selling methodological snake oil. Remember, the term "pollster" is derived from the term "huckster".
Journalists are no mere dupes in this process. We've also seen a disturbing trend of late in which questionable polls find their way into an outlet’s coverage because they appear to match an editorial line, or present a counter-intuitive perspective. After all, if a poll is wrong it’s easy to throw the pollster under the bus and walk away with clean hands.
All of this MUST stop. We are distorting our democracy, confusing voters, and destroying what should be a source of truth in election campaigns - the unbiased, truly scientific public opinion poll.
To be clear, this is not about banning media polls during election campaigns. That would just take us back to the old days of backroom boys leaking false polling, and to the practices we see in less stable democracies around the world. What we need is better, more informed reporting of polls. Here are six easy rules to get us started.
1. IVR polls (robo-calling) are NOT telephone polls. They are tremendously biased in terms of sample coverage. In the last federal election IVR polls were MASSIVELY off on the final vote. Why? Because they systematically under-represented the Tory vote. But, these same pollsters have picked up again in this election without skipping a beat. Here's the question you should be asking them - show us your UNWEIGHTED results. That's prior to adjusting for both demographics and political support. You'll be surprised by what you see.
2. The same question should be asked of pollsters using on-line methodologies to predict vote. Ask to see their results prior to all weighting. You will find that some heavy thumbs are being applied to adjust for under-represented voting groups. While the weighting can produce very good results, it really amounts to no more than an educated guess. And, if that's the case, the results should be reported as such.
3. Disclosing margin of error and the questions asked doesn't represent meaningful disclosure: a rogue poll, a bad poll, and a good poll all look the same on these points. Be honest when something looks dodgy - either don't publish it, or publish it with an editorial disclaimer. Huge shifts in public opinion, even in an election, are the exception, not the rule and should be treated as dubious until confirmed by other polls.
4. A moratorium should be placed on all "new" polling techniques until the pollster has tested them in parallel with more traditional polling methods with a record of success. This should have absolutely been the case with IVR in the federal election. It's a pretty minimum standard to be right at least once with a new methodology before you get to lead a newscast or get the front page headline in a newspaper.
5. The conventional election scenario that's tested in a vote question is - if you had to vote tomorrow, how would you vote? If your pollster asked something different, even if the poll was properly done, it should NOT be reported as "current vote". It is misleading to do so.
6. Spend some time with pollsters. Not just the one you know or use, but with the ones you're hearing about. They are NOT all created equal. Find out about their business. Is this a one person show that subcontracts all of their data collection, and only shows up at election time? What research are they doing on research? What resources do they have available - is it one way for everything, or do they have the resources to adjust depending on what's needed? What are their motives? Are they just publicity hounds trying to promote their business, or are they serious researchers with both the desire and capabilities to perform at the highest level? Shouldn’t you do at least this minimal background check on whomever you are trusting with your lead story?
These six rules require journalists to do two things - kick the tires before publishing a poll, and make it harder for bad or misleading polls to get published. That's the way it's done in jurisdictions that take polling seriously. It's sad that this isn't the case in Ontario today.
Darrell Bricker and John Wright lead all of Ipsos Reid’s political polling for the media in Canada. They have done polls in every major election in Canada since 1988. Ipsos Reid is Canada’s largest market research firm, and is owned by Ipsos, the world’s third largest market research firm.
