# Explaining probability theory versus statistics

I'm not sure whether this question was asked before, but it's hard to search because of lots and lots non-descriptive titles like "statistics and probability".

The context: There is an anecdote I know on the basic difference between probability and statistics:

In an exam in statistics a student would sit near a desk and a professor would begin as follows.

"Let's assume that this is a fair coin." said he, and then threw it, and when it fell onto the desk, he covered it with a sheet of paper so that neither of them would know whether it was heads or tails. "What is the probability that it's heads?" the professor asked.

"It's $\frac{1}{2}$." answered the student.

"No, it's $1$ or $0$, but we don't know which." replied the older man and so the exam ended.

The question(s):

What are your thoughts about this? Does this catches the essential difference? Do you know any other examples that would distinguish in elementary terms between the two (i.e. the probability theory and statistics)? The target audience is math-related fields.

It might be hard to distinguish between opinion and an experience, but I'm looking for various perspectives, so I'm fine with non-concrete answers (e.g. the same story wouldn't work for all the students).

• Not sure, but I think we should call this Schrodinger's coin. – Doc Dec 3 '13 at 20:10
• I'll take a serious stab at this. It seems that we are distinguishing between the probability that an event will occur with the probabilty of an event that has already occurred. Frankly, I'm not sure which of these should be regarded as probability and which as statistics. However, in statistics one looks at available data, so I would judge that the statistical point of view leans toward past outcomes. – Doc Dec 3 '13 at 20:14
• Doesn't this assume that you can't take a Bayesian approach to statistics? – Robert Mastragostino Dec 3 '13 at 20:18
• Well, I guess that the point of this story is that after the coin fell, then it is already determined whenever it is heads or tails, so it cannot have any other probability than $0$ or $1$ (e.g. with multiple worlds hypothesis it already happened and the only thing that is left is determining which world is the one that we live in). However this is slippery, and that's why I'm asking math.SE. – dtldarek Dec 3 '13 at 20:23
• I see probability as a theory of "data creation", and statistics as a theory of data consumption. In statistical applications, the two are rather intertwined, since a statistical approach would typically try to explain the mechanism by which the data being analyzed was created. These explanations necessarily have to be constrained by the laws of probability, and unless assumptions can be made, only the laws of probability can be used to test the statistical model. – nomen Dec 7 '13 at 19:31

As an applied statistician, I have also struggled with the Bayesian/Frequentist issue towards probability. I have come to think of it is as follows (note this is personal view, I am not implying it is the only way to view this):

To me, the essence of the difference appears to be distinguishing between uncertainty vs. variability. Of the two, I see variability as more fundamental, as it is what induces uncertainty. It can do this in at least two different ways: Either there is inherent varability or randomness in what you are investigating (e.g. weather), or there are multiple plausible scenarios that comport with the available data (i.e., many to one relationship), even if the process is completely deterministic.

Variability as randomness appears to be relatively uncontrovesial, under both Bayesian and Frequentist viewpoints. If you flipped a coin many times, you would observe variation in the outcomes while the relative ratio of heads to tails would converge to 1 as your sample got bigger (almost surely...for the probability sticklers out there).Both Frequentists and Bayesians would understand this as between-trials variability.

So, the crux of the debate really centers around variability as uncertainty, or the problem of ranking scenarios that all could have produced the same data. Bayesians choose to use probability as the metric for ranking different possibilities under uncertainty. However, uncertainty could also be quantified using a weaker quantity called a likelihood, which merely compares the probability of the data under two different hypotheses. So, for example, if we flip a coin 10 times and get 5 heads and 5 tails, the most likely value for the probability of heads is 0.5, which represents the maximum likelihood estimate for that probability. It does not take into account any prior data on the coin (if it exists) or the investigators beliefs about how likley, a priori, the coin is to be fair...in other words, the contextual data.

The Frequentists, at least the die-hard types, deny the validity of assigning a probability to an outcome that has already occurred, other than 0 or 1 with no way to determine which is the case. This is the position of your hypothetical professor. Only probability as defined as relative frequency under repeated trials constitutes probability to them.

In my opinion, the entire debate about what is probability boils down to a verbal dispute. I am not trying to trivialize the issues, as verbal disputes are often very important, as they are here, but it is essentially a problem of definitions. The achilles heel of the frequentist position is the "one-time" event, where repetition is not possible. The weakness of the Bayesian approach is one of calibration...there is no objective meanining to a 95% credible interval, or to the Bayesian statement "There is a 95% probability that the hypothesis is true". How much risk are we taking by not accepting the hypothesis? How often would a Bayesian be wrong for this type of experiment?

So, how do I resolve this, at least for my own sense of coherence? I recognize that this is a verbal dispute and realize that the goal of the frequentist concept of confidence is very much aligned with the goal of the Bayesian notion of probability. Some statisticans will dispute this due to mathematical differences in their calculation, which is a valid criticism, while others support it at a conceptual level. For both confidence and Bayesian probability, we are trying to attach a measure of uncertainty to a hypothesis or bracket the true value with a certain degree of certitude.

However, whilst the confidence interval has a clear interpretation in terms of risk of being wrong, the same cannot be said for a Bayesian "credibility interval". As a concrete example, a 95% confidence interval will cover the true value in 95% of the samples where it is constructed. So, you can think of a specific interval as an "oracle" that historically has been right 95% of the time under the same experimental conditions (e.g. if a weather forecaster correctly predicts rain 95% of the time, then you would heed their advice and bring an umbrella with 95% confidence that it will in fact rain). A 95% Bayesian credbility interval, conceptually, provides a range of outcomes that we feel captures 95% of the probability, construed as an abstract measure of support. However, if the same experiment, with the same prior, were repeated multiple times, then it is not a general rule that 95% of the credible intervals will contain the actual value. The error rate may be higher or lower. So, we cannot attach an objective interpretation to such intervals. A good related thread to this is: https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval

Therefore, I see three possible uses for probability, falling along what R.A. Fisher called the "ladder of uncertainty". The first, and most direct, is for when you are dealing with events. In this case, you have random variables that you have some data on and you want to get the probability of next set of observations or calculate the probability of an unobserved event (e.g., medical diagnosis via testing...you know test results and incidence rates, but you cannot observe disease variable)...Bayes theorem is the clear choice in this type of calculation.

The second "rung" is for estimating parameters or statistical functionals under well-defined sampling conditions, where actual or hypothetical repetition is envisioned. A good example is flipping a coin many times to estimate the probability of heads. In these cases, probability as confidence is applicable, as you can assign an objective probability to the interval in the "repeated error rate" sense. An exception to this is where there are multiple ancillary statstics that are relevant to your estimate and it is not possible to determine which one to use. This is called the "relevant subsets problem" and it presents an additional nuance to the use of confidence intervals. See a good article on this here - however, in 99.9999% of analyses, these nuances are ignored and the usual notion of unconditional confidence intervals (e.g., t-test confidence interval) are used. Here, I see Bayesian credibility intervals as inferior subsitutes to the more precisely defined and objective confidence intervals...regardless of the possible paradoxes that may arise in some cases (mostly contrived).

Finally, we get the the third "rung", where we are estimating parameters when there are "relevant subsets", multiple/missing ancillary statistics, poorly performing confidence intervals (e.g., coverage is too conservative relative to nominal value...think Klopper-Pearson intervals), or lack of an exact confidence interval due to parameters not being orthogonal. In these cases we have two choices: either resort to approximate confidence intervals based on appeals to convergence or numerical simulations OR, use a subjective measure of uncertainty and give up on knowing the "repeated sample error rate". I usually go with approximate intervals/confidence as it is generally more accepted in my field. However, the other option is to use Bayesian "probability" or the likelihood ratio to provide a defensible range of possible parameter values given the data. In this regard, I usually use the likelihood approch, as it allows you to incorporate past data and communicate uncertainty without implying that you actually know the "error rate" of your estimate, an impression that may be given by using a Bayesian probability in this instance.

My choice of confidence and likelihood vs. bayesian probability is not based on technical "superiority" but on ease of communication...a probability, in my mind, implies or suggests a rate of error while likelihood communicates plausibility but not an error rate, so its nice to have three distinct terms to refer to: probability, confidence, and likelihood..each based on a repeatedsampling notion of probability, but abstractly implemented in some cases.

Sorry if this got long winded, but its a very interesting philosophical issue that I have worked with for some time and it has practical implications. So, based on the above, you could say that the student and professor are using the same word to refer to different notions: The professor is asking about the "direct probability" of the event "The coin shows heads" while the student is expressing a confidence level in his guess that it shows heads. Frankly, the student's response is both more useful and less pedantic than the professor's...imagine telling a client that something either did or did not happen and you can't (or WONT) chime in on how to weight each possibility...that is both useless and not good for your career.