In 1948, the American writer E.B. Write opined that “the so-called science of poll-taking is not a science at all but a mere necromancy.” While political scientists might disagree, they probably would keep their objections quite right about now. It is well known that expert predictions of the 2015 General Election were off target. But how bad were they and what might they mean for how we think about “data journalism”?
For almost 20 years I’ve studied and evaluated predictions, on subjects as varied as global sea level rise, hurricane damage, the English Premier League table and the quadrennial World Cup. The graph below shows the performance of the forecasts for the two biggest parties, the Tories and Labour, which were the only parties which had a realistic chance of forming a government. The x-axis shows the total number of seats that the forecast missed the outcome by, as compared to a simple baseline expectation based on the March, 2015 composition of Parliament. The RED bars indicate a performance worse than the naive baseline, and BLACK bars indicate an improvement, and no, there are no black bars on this graph. (You can see the details of the forecasts and the evaluation methodology at this post.)
It is difficult to describe this graph as anything other than mass carnage for the forecasters. The predictions were off, and not by a small amount. Nate Silver, known for his accurate predictions of the past two US presidential elections and his website FiveThirtyEight, toured the UK before the election. During his visit he opined to the British public, “What we know is that it’s highly likely you won’t have a majority.” Well, no.
We can expand the evaluation to include predictions for the Liberal Democrats, UKIP and Scottish National Party. (Note: Only 8 of the original 13 forecasts included LD, UKIP and SNP; a graph of these results, as well as one for just the Conservatives, Labour and the Liberal Democrats can be found here.) With the SNP revolution occurring in Scotland, we would expect that this would improve the forecasts, since the naive baseline had only 6 SNP members in Parliament. (UKIP turns out to be mathematically irrelevant in this exercise.) Even so, adding in the other three parties only raises four forecasters above the naive baseline. It is worth noting that the worst performing forecast method overall had the very best prediction for the number of SNP seats.
Overall, if we take the set of forecasts as an ensemble and ask how they did collectively (simply by summing their seat errors and dividing by the number of parties predicted), the picture remains unimpressive:
- Two-Party Forecasts (13 forecasts): degraded from Naive Baseline by ~38 seats per party
- Three-Party Forecasts (12): degraded from Naive Baseline by ~17 seats per party
- Five-Party Forecasts (8): degraded from Naive Baseline by ~0.3 seats per party
So what lessons should we take from this exercise?
One lesson is that while predicting elections is interesting, fun, and perhaps educational from an academic perspective, it may not add much to our democratic practices. Nate Silver, for better or worse, has become the face of poll-driven “horse-race journalism” in which political and policy context is mostly ignored with the numbers pretty much all that matters. This is of course ironic, because Silver used to complain about punditry and horse-race journalism. Yet during his recent promotional tour of the United Kingdom he did so as a celebrity, pontificating on the horse race. Not discussed by Silver were questions about subjects such as the future of the NHS, options for recharging UK productivity, or the desirability of Scottish independence or a possible EU referendum.
My criticism of election forecasts goes back a long way, in fact all the way back to my PhD studies in political science in the early 1990s. Back in 2004 I wrote: “Rather than trying to see the future, political science might serve us better by helping citizens to create that future by clarifying the choices we face and their possible consequences for policy.” In treating politics like a sporting event, we diminish the partisanship, the choices, and the fundamental values that lie at the core of politics. I fear that data journalists have diminished our politics.
A second lesson is that we often forget our ignorance. Back in 2012 Nate Silver asked whether political scientists could predict electoral outcomes with “amazing accuracy.” His reply was on target: “The answer to this question, at least since 1992, has been emphatically not. Some of their forecasts have been better than others, but their track record as a whole is very poor.” The 2015 UK General Election reminds us of this fact, as it has the folks at FiveThirtyEight. Sure, it does seem possible to anticipate US elections, but this may say something about American exceptionalism (e.g., highly partisan with well-gerrymandered districts, with a relatively simple electoral system that is overwhelmingly well-surveyed) rather than anything about the predictability of politics more generally.
I don’t mean to pick on Nate Silver (Disclaimer: I worked for him briefly in 2014, and admit to sometimes being seduced by horse-race journalism. In addition, the FiveThirtyEight forecasts were actually produced by a UK-based team of academics.) I have long had a lot of respect for Nate, not least because in the US at least, he figured out how to systematically integrate and evaluate polls, something that academic political scientists utterly failed to do.
At the same time, his influence in the UK elections – along with that of other, less visible, forecasters — is significant enough to warrant critique. Here is one example of the overwhelming influence of a dominant “narrative” in popular discourse. One pollster, Survation, conducted a survey before the election that in hindsight proved remarkably accurate. But they chose not to publish. Why the self-censorship? Damian Lyons Lowe explained that “the results seemed so “out of line” with all the polling conducted by ourselves and our peers – what poll commentators would term an “outlier” – that I “chickened out” of publishing the figures – something I’m sure I’ll always regret.”
While the Survation team has to live with the decision not to release their poll, I can understand the pressures that exist not to contradict popular narratives expressed by loud and powerful media bodies. These pressures can mean a public discourse characterized by narrow perspectives that exclude other, off-narrative expert voices. Sometimes, however, the popular narrative is wrong.
The role of data journalists and their close cousins, explanatory journalists, should not be to limit public discourse, either intentionally or unintentionally by weight of influence, but rather to open it up. This means going beyond the numbers and into all the messiness of policy and politics. Punditry absent rigorous evidence and data is impoverished. But data and evidence absent political and policy context is impoverished too. Data journalism, like our democracies, remains a work in progress.
Note: This is based on an earlier version, first posted at http://rogerpielkejr.blogspot.com.
guardian.co.uk © Guardian News & Media Limited 2010