How do I evaluate the accuracy of sport trade probabilities? Recently, a sports broadcaster gave probabilities of certain trades occurring before the 2015 NHL trade deadline:

I've seen unsubstantiated sports claims before, but this one struck me as interesting due to the granularity of the estimates (rather than just saying likely/unlikely).
After the trade deadline, each of these events will have occurred, or not occurred.  How would a statistician evaluate the quality of the estimates (in hindsight)?
For example, I know that I could take all of the "30%" estimates, and over an infinite sample, if they were accurate, it should converge on 30% of the events occurring.  But I don't know:


*

*How to reconcile this with a variety of probabilities.

*How to put a number on the "quality" of the probabilities (I'm assuming this will be paired with some confidence metric).


To extract the data from the image, here is what we're working with:
P(CF) = 0.85
P(DW) = 0.85
P(KH) = 0.75
P(TB) = 0.6
P(JL) = 0.6
P(MS) = 0.6
P(JG) = 0.3
P(JR) = 0.3
P(NK) = 0.3
P(DP) = 0.2
P(PK) = 0.2

 A: Under the hypothesis that the probabilities are correct, the total number of correct predictions will follow a Poisson Binomial Distribution. It's not a nice function to work with, but we can get a rough approximation using the mean and variance, which we can calculate:


*

*The expected number of successes is just the sum of the probabilities: $5.55$

*The variance is just the sum of $p_i(1-p_i)$: $2.11$


We can try to approximate this by a normal distribution, in which case, we would conclude that there are errors in the predictions with $95\%$ confidence if the number of successes, S, falls outside the following interval:
$$5.55\pm1.96\sqrt{2.11}$$
Roughly speaking, it would be highly unlikely for us to see greater than $9$ or less than $3$ successful predictions, assuming that the prediction probabilities were accurate.
Quality of Predictions
I'd like to suggest two different measures for accuracy, then combine them:
Lets take the example where we give every prediction $i$ is expected to have a $\rho_i$ chance of success. The weakest prediction one can make is to assign $p_i=\rho_i$, since it is perfectly along expectations. Conversely, predicting something completely opposite to expectations is a very strong prediction. So, a metric for the strength $S$ of each prediction can be:
$$S_i=\frac{1}{\max\{\rho_i,1-\rho_i\}}|p_i-\rho_i|$$
Overall strength of a group of predictions can be summarized by the mean strength:
$$\bar S:= \frac{1}{N}\sum_{i=1}^N S_i $$
Ok, so that takes care of the strength of the predictions, now for the accuracy. Again, Ill start with a single prediction. Lets call our metric the Deviation ($D$) given the observed outcome $O_i$ (i.e., success or failure):
$$D_i:=|O_i-p_i|$$
And, similarly to the strength assessment, we can use the mean deviation as our aggregate metric:
$$\bar{D}:=\frac{1}{N}\sum_{i=1}^N D_i $$
Now, we have a two-dimensional assessment of the prediction: its "strength" or a priori confidence, and its relative accuracy.
In the case of $50\%$ prediction across the board, the mean deviation will be $0.5$ but the mean strength will be $0$. Moderately wrong, but useless predictions.
In the case where each prediction is $0\%$ or $100\%$, the strength would be $1$ and the mean deviation could be anywhere from $0$ to $1$, depending on how well they did. So, if they are spot on, you would have very strong, very accurate predictions, or if they are way off you would have very strong, but very wrong predictions.
Combining these two, we can create a strength adjusted accuracy $A$:
$$A:=\bar S\times (1-\bar{D})$$
This measure combines the two ideas, favoring predictions that are both strong and accurate, and discouraging overly conservative or overly radical predictions.
