| Dear Readers: One of the surprising late developments in the 2024 campaign was the final Des Moines Register/Mediacom Iowa poll, which showed Kamala Harris narrowly leading in Iowa. That poll, though a telling indicator in the past, ended up dramatically underestimating Donald Trump in that state. In the piece below, a group of scholars from the University of Iowa report on how the poll may have impacted voter expectations about the election and assess the poll (and offer some observations about challenges of polling in general).
— The Editors |
KEY POINTS FROM THIS ARTICLE
— In the days leading up to the 2024 presidential election, the Des Moines Register/Mediacom Iowa Poll, spearheaded by pollster J. Ann Selzer, showed Kamala Harris leading by 3 points in Iowa.
— Besides receiving national news coverage, this poll had a significant impact on the expectations of Iowans themselves.
— Such dramatic public findings, which this Iowa Poll represents, can mislead voters and undermine trust in the polling enterprise itself.
About that Iowa poll
On Nov. 3, the front page of the Des Moines Sunday Register emblazed banner headlines: “HARRIS NOW LEADS TRUMP. She pulls ahead 47% to 44% among likely Iowa Voters” according to their Iowa Poll, celebrated since 1943. This news spread quickly throughout the media, with Des Moines Register and Ann Selzer becoming notable “search” terms in 38 and 50 states, respectively, overnight. The excitement was captured in a New York Times lead the day after: “What That Surprising Iowa Poll Might Be Telling Us.” In the actual vote, on Nov. 5, Donald Trump carried Iowa by 13 percentage points, for a total forecasting error of 16 points. How did this happen? What were the effects?
The notion that published vote intention polls influence voter behavior expresses an hypothesis of long standing. Normally, it would be difficult to trace out the effect of this last minute, single poll on would-be voters in a single state. However, we had in the field our own poll of Iowa voters, namely a sample of 214 University of Iowa students. The respondents were asked a series of questions about the upcoming presidential election, including which candidate “will win the most votes?” and which candidate “will win the Electoral College?” (In a key reported demographic, the partisanship of the students was virtually an even split, 40% Democrats and 41 % Republicans.) With respect to the first question, 64% said they expected Kamala Harris to win the popular vote, compared to only 36% for Trump. With respect to the second question, only 45% said they expected Kamala Harris to win the Electoral College, compared to 54% who expected Donald Trump to win. Thus, in terms of actually claiming the office of the presidency (via the Electoral College) the students foresaw a Trump victory nationwide.
Was the distribution of these Electoral College expectations influenced by the announcement of the Harris rise in the Iowa Poll? It appears so. As luck would have it, the student survey stretched from Nov. 1 to Nov. 4, thereby bookending the first public appearance of the Iowa Poll on Saturday, Nov. 2 at 6 p.m. central time. In addition to the above questions, the students were also asked which candidate “will win the Electoral College vote in the state you live in?” Thus, we have a ‘natural’ field experiment, with a pre-treatment group (who answered the survey before the Iowa Poll announcement), and a post-treatment group (who answered after). The results show a statistically significant 9-percentage-point rise (p < .05, one-tailed) in expectations of a Harris victory among those who said Iowa was their home state (150 respondents). While these data are observational, they are nevertheless highly suggestive. It looks like the Iowa Poll did, in fact, increase Iowa public opinion favoring Harris to win the state.
This finding has special relevance, given this Iowa poll was a gross outlier, off the mark by 16 points. Why? The poll, carried out by telephone interviews Oct. 28-31, utilized a sample of 808 likely voters (randomly drawn from landline and cellphone numbers supplied by Dynata), with a theoretical margin of error of only 3.4 percentage points. In the aftermath of the real, and large, error, the newspaper conducted its own review of what went wrong. They concluded “the weighting in this poll was minimal and yielded a final sample of likely voters that matched the percentage of the most recent poll.”
Ann Selzer herself “has not found voters’ recollections of their past votes to be highly reliable,” – a weighting technique that became more commonly used in 2024 polling. However, in her own technical review of the poll, she actually reports results weighting on voter recall. This adjustment for respondents, via their presidential vote in 2020, yields a strong forecast of a Trump victory, with a plus 6 percentage point lead over Harris (see the table on page 11 of her review). Such evidence suggests this increasingly common weighting practice would have produced a more accurate result, albeit one that still understated Trump’s eventual winning margin. Of course, that was likely not the sole problem. Selzer did weight on the demographics of age, sex, and congressional district, but other markers, e.g., rural-urban, could be considered as well.
Still, Selzer’s reputation as an outstanding pollster cannot be denied. For example, The Guardian newspaper, heralded her as the “Queen of Polling”. Since 1987, she has managed most of the Des Moines Register election polls and scored noteworthy successes with final results. She alone foresaw Barack Obama’s 2008 triumph in the Iowa caucus, called Joni Ernst (R) in a pivotal 2014 Senate race, and predicted Trump’s Iowa general lead over Hillary Clinton in 2016. Regarding methodology, she said after another successful 2020 general election poll: “I assumed nothing. My data told me.”. Still, she had misses. In 2004 in Iowa, she predicted John Kerry over George W. Bush. More recently, her final 2018 poll showed a Democrat narrowly ahead for governor, but Republican Kim Reynolds won. But these could be seen as cherry-picked examples. Instead, we can look at gathered final Selzer poll data from 34 races (president, governor, senator) from 1996-2024—most of these are from Iowa, although some are either national polls or from a handful of other states (you can see the polls on Wikipedia, although we also checked them against FiveThirtyEight’s database). . In each of these races, the Republican and the Democratic vote shares are forecast, making for a sample of 68 estimates. Suppose, by way of summary, a somewhat generous margin of error for each estimate, at 4.0. We can then assess how many estimates fell beyond that expected margin. We observe 11 beyond-margin errors for the Democrats, and 17 for the Republicans. Overall, out of the 68 estimates, 28 show beyond-margin errors, meaning that about 41% of the forecasts demonstrated more than the expected amount of error (i.e., 28/68 ~.41).
These results, admittedly partial, point to underlying problems in the methodology that merit further exploration. It must be mentioned that, while the Iowa Poll rests under the microscope here, such difficulties, over the past two presidential elections in particular, have been rife in the polling world. Take 2016, where the polls failed to predict the Electoral College victory of Trump. For 2020, one of us (Michael Lewis-Beck) along with Natalie Jackson even found “the polls performed worse than in 2016.” Examining 322 national polls, they reported the margin of difference between the poll result and the actual 2020 result was, on average, almost 5 points in the Democratic direction, with respect to the two-party popular vote.
Did this Democratic bias operate in 2024? On the eve of the election, scholar Claire Durand found that 159 national U.S. presidential polls, from Sept. 1 to Nov. 3, revealed the vote intention plot lines of Harris v. Trump support steadily converging, until the graph “shows a perfect equality,” at a 50-50 popular vote split. Based on nearly final numbers, it appears that Trump won the national popular vote by about 1.5 points, so the national polls were fairly close to the actual outcome, albeit underestimating Trump again. To the extent aggregate polling outcomes reflect an actual closeness, malaise surrounding the polling enterprise may begin to lift. The scramble by pollsters to improve their methods, e.g., more mixed mode, more random sampling frames, may be showing an effect. However, we must be cautious. A skeptic might say that the underlying problem has not gone away. The population lists necessary to execute properly a scientific random sample are evanescent. The list of voters does not exist before the election, because they have not yet voted. The goal, a la polling scholar Leslie Kish (writing way back in 1965), is to arrive at a list of respondents who, operationally, represent a valid enumeration of the voting population under study.” A necessary goal, but one not easy to achieve.
| BIO INFO HERE |