In 2012, many polls predicted Mitt Romney, the Republicans’ nominee, would defeat President Barack Obama for the presidency. Well, it went slightly different though. The Brexit was a surprise not only for British citizens but for all Europe, against all predictions from specialists and surveys agencies. And I don’t want to mention (but nevertheless I do it) the incredible worldwide shock of US elect president Trump overcoming all the forecasts seeing Clinton far ahead by numbers. Three different public-opinion polls, three important elections, three decisively erroneous results. Once a seemingly infallible cornerstone of the political system, public opinion polls have racked up a few big-time fails in recent years, embarrassments that compelled a leading firm to conduct an internal audit to find out what went wrong. Analysts are also openly questioning whether the industry, whose leaders and techniques are from the 60s, has kept up with a rapidly transforming, highly-mobile electorate – one that’s relying on everyday technology to opt out of the public discourse.
“The science of public surveying is in something of a crisis right now,” says Geoffrey Skelley, a political analyst at the University of Virginia’s Center for Politics.
And it matters because “polling is a very important element of democracy,” said Michael Traugott, a University of Michigan political science professor who specializes in polling and opinion surveys. Traugott also helped prepare a groundbreaking report on how Gallup, a public-opinion leader, erroneously predicted Romney would defeat Obama in 2012.
Polls “give the public an independent voice that’s not generally present” otherwise in politics and political news coverage, Traugott said. But he says the recent errors, and a steep decline in the number of people responding to opinion surveys, is “a worrisome trend because one of the main claims of polling is that it represents the people’s views.” In an unprecedented internal report on its 2012 Romney blunder, Gallup says it made mistakes in its core samples, including its racial makeup and political ideology, as well as its overall methodology. Gallup’s audit, however, also says the entire industry is due for an overhaul, with some of the leading firms using analog, black-and-white methods in a digital, multicultural world. Case in point: the rise of the cell phone and the fall of public engagement in opinion surveys. Besides not being tied to a fixed address – it’s not unusual for owners to have a different area code than where they actually live, and the numbers usually aren’t listed in the white pages – cell phones provide more control over its users’ privacy than a landline. It’s likely, analysts say, that the ability to screen or block incoming calls has accelerated the public’s unwillingness to take part in what used to be considered a civic duty.
“Everyone in the industry is worried about the falling response rate,” Tourangeau says.
Traugott says the percentage at which people participate in opinion polls has bottomed out in the past few decades, from more than half in the 1980s to the single digits today, and most experts believe cellphone use is the reason.”It’s not just an American problem. it’s a worldwide problem,” Tourangeau says. At the same time in the U.S., a federal law designed to protect consumers from aggressive debt collectors or telemarketers bans pollsters from using automated calls to get opinions, even if it’s on important issues like the presidential election. “People are leading more active lives, and they’re harder to locate,” Traugott says. People would rather text, make calls or perhaps play another round of Candy Crush Saga than spend up ten minutes or longer “for an interview with an organization they they might not know and a survey whose content might be unclear,” he says. Other factors, Tourangeau says: Gated, private communities that door-to-door surveyors can’t reach, and more survey subjects who don’t speak English as a first language. Celinda Lake, a pollster, political consultant and president of Lake Research Partners, a Washington, D.C.-based polling firm, says polling has seen “kind of a steady decline. It’s getting harder to reach people. It’s also harder to get them to cooperate.”
That means Lake Research, Gallup and others spend far more time, effort and money than ever before, trying to get solid opinions and deliver an accurate snapshot of the public mind. “You try them more often. We’ve upped the number of callbacks; we used to do two or three. We now do three, four and five,” Lake says. Sometimes, she adds, they even make appointments with survey subjects to get their participation. But not every pollster has the time, the money, or a staff big enough, to up their game and dig into a major opinion survey with that level of commitment. That’s especially true when campaigns, polling firms and news organisations are competing for attention in a hyper-speed, social media-fueled, 24-hour news environment.
“That means there’s getting to be a broader and broader range of quality of polls, where some people have the resources and some don’t,” Lake says. “And it makes for more variability” in across-the-board quality of results on a particular issue or political campaign. Another key factor is polling firms’ methodologies, the “secret sauce” of the industry. Put simply, if you ask the right sampling of people the wrong thing, you’ll get a bad result. Ultimately, Traugott and others say, the industry is responding with better methodology, improved modeling of public behavior, smarter ways to reach people (including Internet solicitations and small amounts of cash) and a commitment to learn from its mistakes. And they each said the industry as a whole nearly always gets it right.
“The trends are tough, Tourangeau says. “But it’s not like the whole field has collapsed either. I’m not ready to give up on surveys at all. It’s an essential tool for the government and the politicians,” he says, “and it’s not going away.”
Well, surely, we will continue to use those tools and polls all around but we do not know whether to rely on them or to believe them. A big change is needed. For the rest, some big changes already started.