Nassim Taleb has tweeted criticism about 538's political predictions that I don't follow. He seems to be making a theoretical argument. I am very weak on stochastic modeling (though I know basic PDEs). Can someone help me understand at least the gist of what he is saying?

enter image description here


Taleb's example chart is not particularly helpful to his presentation, as it suggests that FiveThirtyEight were always closer to the final result than his preferred methodology. In fact the two were estimating different things, and the chart demonstrates this with 538's much greater volatility.

Taleb seems to be saying that the assessed probability X wins the actual election (as seen $100$ or even $5$ days before the election) should be much closer to $50\%$ than FiveThirtyEight suggest, as lots of different things could happen to shift opinion in either direction between now and election day. He uses the mathematics of options for this, including the "no arbitrage" condition, but that detail is not particularly relevant.

FiveThirtyEight's methodology essentially takes current information available, especially but not exclusively from recent opinion polls, and projects uncertainty around this information taking into account other factors such as demographics, endorsements and polling trends, and then runs this uncertainty though an electoral college system to translate into uncertainty over the final result.

Taleb's analysis includes the possibility that things will change and that this could change people's opinions in a way not covered by FiveThirtyEight's look at current opinions. So while 538 concentrates on current uncertainty, Taleb's analysis is dominated by future uncertainty. If 538 says Clinton currently has a $74.1\%$ chance of beating Trump, then Taleb would reply that this is excessive (he would say the number should now be closer to $50\%$ as opinions could change between now and November), to which 538 would respond that their number was an assessment of the current position given their best assessment of voters' current opinions and the uncertainty about reports of the current position.

Taleb has bolded the word future to highlight this difference.

  • $\begingroup$ Thanks! So does this boil down to a disagreement about how to measure variance? The two graphs seem like they could be simply rescaled from one to the other. $\endgroup$ – abnry Sep 1 '16 at 17:10
  • $\begingroup$ @abnry You could say that Taleb adds in extra uncertainty (not quite variance), which only declines substantially just before the election day. But even Taleb's method takes account of the polls, which is why local maxima and minima often match on the two lines $\endgroup$ – Henry Sep 1 '16 at 18:29
  • $\begingroup$ +1..What about the 538 Polls Plus forecast? It attempts to account for these dynamics too, where it shows the prediction quite a bit closer than the polls taken at face value. Also, I believe the polls are weighted by predictive accuracy as well, so it's not as if they are oblivious to the fact that they are predicting a future binary outcome using existing poll data. $\endgroup$ – user237392 Sep 2 '16 at 1:37
  • $\begingroup$ @Bey 538's forecasts in general take into account different polls' variances and biases, and their Polls Plus forecast takes into account things like the demographics, endorsements and polling trends I mentioned, but these are all adjustments for current uncertainty about recent polls reflecting current opinion, not future uncertainty about opinions changing between now and election day $\endgroup$ – Henry Sep 2 '16 at 7:20
  • 1
    $\begingroup$ I don't think they are simulating the outcome as if the election happened today. See here: fivethirtyeight.com/features/… (especially step 5). Yes, they do use polls to develop forecasts but the polls are assessed based on their predictive accuracy -- how well do the polls predict the Election Day result...this is decidedly a future oriented estimate. So I don't see how Taleb can accuse them of not taking future uncertainty into account-- it will be baked into the poll's predictive accuracy. $\endgroup$ – user237392 Sep 2 '16 at 16:32

In short Taleb argues that Silver's predictions are too volatile to be reasonable. An extreme example of this principle is that if a year from the election an outcome has a 95% probability then you should be unlikely to have an event that would drop the probability to 50% the next day; if such an event were likely you would be assigning a probability closer to 50%. While such swings could happen occasionally they should not happen repeatedly.

It is likely that Nate Silver's models are too sensitive to information well in advance of the election; perhaps having a stronger prior close to 50% more than a couple of weeks from the election is more reasonable.

That said Taleb demonstrates this for one highly idealized model--a Brownian motion. The real process has discrete shocks at predetermined times, the debates. It also has randomly timed shocks such as Comey's statements. I suspect even a model that accounted for those would have been suggesting Nate's forecast should have been less volatile . Also the only prediction I heard from him was a statement that a Trump Presidency is very possible---qualitatively similar to Nate Silver's result .


I think it's fair to say 538 is much more answering the question: "what would happen if the election were tomorrow?" and that is a pretty good interpretation of what polls do. That's not the same as the question, "who is going to win on the actual election day?" which is, to be sure, less certain. 538 does present these as forecasts, and there's a fair point that this isn't what it is.

However I think humans intuitively understand polls to be a measure of 'now', and I suspect its actual meaning is what readers expect. That is, most don't seem so very surprised in changes in the 'forecast'. Or: if the answer to "who is going to eventually win?" really is "about 50/50" the whole time, then that's not a very interesting question to answer. Maybe if you're betting on the outcome, but that's not the purpose of 538.

I think this is a long long way to go to dress up essentially an argument about semantics an assumptions in a lot of math. He seems alone in this interpretation of what 538 is doing. I think it's a bad-faith argument from a professor that is attempting to make this look both very difficult and only a math question, when it just isn't so.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.