Tuesday, October 7, 2014

Polling flaws = follow trends, and not only one poll

It’s interesting to watch media try to gauge where things stand for the November elections, not only because lazy horse-race coverage beats thinking and analyzing political positions, but also because the changes of the last few years have made it increasingly tougher to get a read on that horse race from polling data.

Polling in 2014 is as tough as its been in years, as this article by Carl Bialik in Fivethirtyeight.com shows. That’s the feedback Bialik got when he talked to numerous pollsters across the nation. And what’s the biggest problem in getting it right? Figuring out who is actually going to vote next month.
We asked pollsters if they expected more or less error in Senate election polls — the difference between what the latest pre-election polls show and actual vote margins — this year than two years ago. Ten said they expected a higher average error, while just five predicted lower error.

No one cited low response rates as a reason to expect poll error. Perhaps that’s because pollsters have managed to maintain strong national-election records despite declining response rates.

Instead, the top reason cited was the difficulty of forecasting turnout in midterm elections, without a presidential race to bring voters to the polls. And the crucial midterms are in states that don’t usually have close races. “The key Senate battlegrounds this year are also places like Alaska, Arkansas, Kansas, Louisiana, etc., where most of the public pollsters don’t have a ton of experience,” one pollster said. “It’s not the Ohios and Pennsylvanias and Floridas of the world that we’re all used to polling a lot.”
With that in mind, Bialik asked the pollsters which are the places that are hardest to poll? And the answer ranged from high-population places such as New York City, California, Florida, to low-population places like Alaska, Hawaii and Nevada. But it seems there are two commonalities in the places hardest to poll accurately.

1. Heavy amount of the population that has moved into or out of the area, with area codes on their phones that may not match where they live.

2. High amount of immigrants and/or non-whites, which have relatively low response rates, with those who do respond not being a necessarily accurate sample.

If you take this down to Wisconsin, which areas become likely to be underrepresented and/or erratically reported based on this criteria? Madison (lots of college students and young professionals from other states) and Milwaukee (higher numbers of people with mobility, along with have a sizable minority and immigrant population).

Both of these heavier chances for error showed up in the latest Marquette poll, where Dane County’s share of people polled was only 7.9% in the “likely voter” sample that put Scott Walker up 5. This is different than what the 2012 November elections told us, where Dane County represented 10.2% of the vote in 2010 and 9.9% of the vote in 2012. In fact, it is probably more reasonable to assume Dane County will be closer to 11 or 12% of the overall vote in this election, given that where Latinos voted for Barack Obama 71-27 in 2012, and 66-34 for Obama in Wisconsin. Add in the fact that Walker heads up a suburban-based Wisconsin GOP that has promoted voter ID as a way to make sure “non-citizens” don’t get to vote, which I’m guessing is a stance that doesn’t go over very well in Hispanic communities, and I’m more than a bit skeptical that Scotty is leading with Hispanics. As Bialik’s FiveThirtyEight column notes, this likely overstating of WisGOP support from Hispanics in the Marquette Poll is similar to what is experienced in Nevada.
Some pollsters also named Nevada as challenging, saying Nevadan Hispanics and blacks who answer polls tend to vote more conservatively than their counterparts who don’t. Tom Jensen, of Public Policy Polling, said that many Las Vegas residents keep unusual work schedules that make them hard to track down. Another pollster added, “Too much happens under the radar for opinion pollsters to pick up. … Nobody gets elections right in Nevada.”
And this is why I have major issues with lazy Wisconsin media portraying last week’s Marquette Poll as an indicator that Scott Walker has some kind of notable lead in the Wisconsin governor’s race. As mentioned at the time the poll came out, its likely voter sample should not have had a “GOP +4.6” Party ID advantage for a state that has more often than not voted for Democrats in statewide elections in the last 26 years.

It’s not like the Marquette Poll numbers for candidates or Party ID are set in stone, as shown by these results in its likely voter samples from 2012 over the last 6 weeks of the campaign.

Wisconsin poll for president, Marquette Law 2012
Sept 27-30- Obama 53-42, Party ID Dem +7
Oct. 11-14- Obama 49-48, Party ID Dem +1
Oct. 25-28- Obama 51-43, Party ID Dem +5

Final Result- Obama 53-46, Party ID Dem +5

So yes, the Marquette Poll “hit” on their final poll, but unless you think we were amazingly fickle up here in the last month before the November 2012 elections, MU Law was way off with their modeling before that last poll. Just because their accurate final numbers make some people call them the "gold standard" does not necessarily mean they are on the mark all the way through. And I'm not implying that Franklin was intentionally rigging the poll to have the outcomes lean a certain way - it's a hard job to do and perhaps this reflects the responses of people willing to talk about the race at a certain time.

Which is why the aggregation of polls is definitely the way to go to figure out the state of a race (HuffPost Pollster has Walker up 48-47 in this model, using all polls from the last 6 months), because some will slant one way, and some will slant the other. This may be out of malice (especially with GOP pollsters trying to slant media coverage), but it’s also difficult to model what turnout will be in this time of low response rates among certain population. It’s also why it’s hilarious (and likewise scary) to see the hacks at Gravis Marketing backtracking on their poll that had Mary Burke up 5 points a little over a week ago, as they saw the Marquette Poll and panicked, claiming that they “oversampled Milwaukeeans” and made the result too Dem-leaning. Sure enough, they released another poll yesterday that was 50-46 Walker, with a higher amount of white voters listed- I'm sure that's just coincidence.

Given the history in 2012, perhaps Charles Franklin at Marquette makes a similar adjustment over the next couple of weeks, adjusting his poll sample to better reflect the larger and growing population of Dane County, and getting closer to the electorate that will be likely to turn out on Nov. 4, instead of just writing down the responses of who picks up the phone and underweighting certain demographics as a result. This will likely "shift" the race back toward Burke, and then it’ll be interesting to see if certain media changes its tune on the horse race in Wisconsin, and admits we’re back to where we were a month ago- in a basically even race that will likely come down to which party’s voters get their ballots cast (and counted).


  1. But the LV screen in the early September poll had Dane County at 12.3%, the August one at 9.4%.

    Of the RV sample, Dane County was 7.4% in the late September poll, 10.8% in the early September, and 9.5% in August. As of October 1st, Dane County contains 10.1% of Wisconsin's registered voters.

    The Marquette poll weights by geographic region, not county: with a typical sample size of about 800, with a 2σ interval you'd expect Dane to be about 10.1±2.1% of the RV sample. Yes, Dane was notably under-represented in the late September poll, but the impact of this is the difference between voting habits in Dane and the rest of the Madison marketing area (which formed a correspondingly larger fraction of the sample).

    The other counties in the region are Columbia, Grant, Green, Iowa, Juneau, LaFayette, Marquette, Richland, Rock, and Sauk. In the recall they were D+1.3 vs D+38.9 in Dane, so going from a 10.1% Dane / 7.7% rest of Madison DMA actual split to a 7.4% Dane / 10.4% rest of Madison DMA sample split would make an actually D+22.6 (ish) go to D+16.9, some 5.7 points redder.

    But that's just 17.8% of the entire state, so the net effect on the statewide sample is to make it 1.0 points redder, several miles inside the margin of error for this poll. Yes, it's theoretically possible to weight by county instead of region, but in practice it's a really bad idea because that level of weighting granularity will push your error bars way up into the stratosphere (your age/sex/race/location cells wind up with only a handful of people each on average, and a fair few of them will have just 1 or 2 people (with commensurately large statistical errors in the cell) weighted up to many more: weighting amplifies the statistical errors in polls.

    Sometimes Dane County is undersampled, sometimes it is oversampled. That's the nature of polling: you can't delve into quite such detailed weighting without making your results uselessly vague.

    I don't think that's the problem with MLSP: I think its main issue is that the likely voter model hasn't actually been put to a thorough test yet: their 2012 results would not have been affected much at all by including "very likely" to votes alongside the "absolutely certain"s.

  2. Actually, you prove my point Geoff. Little undersamplings here and there add up to a sizable advantage. Which is what infuriates me when I hear TV networks claim "Walker leads by 5" because of that one GOP-leaning poll. It's lazy, and incorrect, but it slants coverage and slants the opinions of certain low-info voters who want to vote for a winner.

    That's why we need to push back on it

    1. Actually, I don't: I pointed out how Dane County was oversampled in the first September poll - this changes from poll to poll. In the late September poll, Ozaukee and Sheboygan counties - both redder than the rest of the region as a whole - were under-represented in the RV sample, while more blue-leaning Kenosha was over-represented.

      Little undersamplings here and there at small levels do not tend to add up to a "sizable advantage", since they are randomly distributed. If I have time I'll go in and reweight for historic redness/blueness at the county level to demonstrate it, but really it's your claim so the onus is on you :)

      The reason pollsters want to avoid making their little weighting boxes too small (in MLSP's case, each age/education/sex/region box - it's a touch more complicated than that but not enough to change the point) is that e.g. Trempealeau County has 0.46% of the state's RV's, but is represented by just a single respondent, 0.125% of the sample. Trempeauleau was slightly red in the recall, almost exactly 50-50 in 2010. You'd expect about 0.23% Burke and 0.23% Walker contributions to the state totals from this county, but instead since there is only a singular respondent you get 0.23% less of one and 0.23% more of the other, nearly a 0.5% error to the final result is introduced if you insist on weighting at the county level, just from this one county. Add up all those errors (variances) from the other 71 counties and you get a less useful end result than if you take larger geographic aggregations.

      Also, my point is exclusively reserved for the underlying RV sample - I've already criticized the LV sample in my previous comment.

      (Whilst I was poking through voter registration statistics, I noted that someone is doing a bang-up job of registering new voters in Dane County in September: there were a net new 4,722 registered voters in the state over the course of September, and a net new 3,360 in Dane County. In percentage terms, only LaFayette and Menominee top it. It's not you by any chance is it?)