4
$\begingroup$

I fully understand the mechanics of Bayes' Theorem. However, I am wondering when do I need to use it? If I am able to compute the posterior probability directly from measured data, why would I need to use Bayes' Theorem?

For example, consider NBA basketball games. Let $A = team\ wins$ and let $B = team\ scores\ 100\ points$. I want to compute the posterior probability $P(A|B)$, or $P(team\ wins | team\ scores\ 100\ points)$. If I expand this out using Bayes, I would get:

$$P(team\ wins | team\ scores\ 100) = \frac{P(team\ scores\ 100 | team\ wins) \cdot P(team\ wins)}{P(team\ scores\ 100)}$$

Suppose I have the entire log of my team's results. I can compute the posterior (left-hand-side) directly from the logs by simply building a contingency table and doing the appropriate calculations, just as I would to compute the likelihood probability $P(B|A)$. Why would I need to compute the posterior through Bayes' Theorem? Would I use it when I have sparse data (e.g. A and B seldom co-occur)?

$\endgroup$
8
+50
$\begingroup$

You are absolutely right in that you don't need to go through Bayes' formula to calculate the relative frequencies -and this is the critical point:if you do as you suggest, you are NOT calculating probabilities, and they are not even "posterior" - they are just "conditional" (not the same thing).

Your question is equivalent to ask "Is descriptive statistics the same thing as inferential statistics?" I know you didn't use the term "statistics" in your question, but you cannot escape it.

Naturally, since you have the data, you can easily compute the relative frequencies, be it joint, marginal, or conditional, of the events that have occurred. What does that tell you? That for events passed, the relative frequencies of the events you are interested in, were so and so.These are not probabilities yet. In order for them to be treated as probabilities, you have to make additional assumptions.

Why? because probabilities are used to describe (and hopefully manage) uncertainty - and uncertainty relates to the unknown (usually the future, but not necessarily - it may refer to events that have happened but for which you don't know the outcome). So in order to move from the known and certain (the empirical frequencies you have calculated -which is what descriptive statistics is all about) to the unknown (the probabilities) it is obvious that you have to make additional assumptions to somehow use relative frequencies in place of probabilities- they are not automatically equivalent.

And here is where "frequentists" and "Bayesians" part ways. Tailored to your question (and oversimplifying of course),

The frequentist would make the following assumption:"I assume that my sample (the data from games played), is representative of what happens "in general" with this team. So, next season will be approximately the same. Given this assumption I can use the relative frequencies obtained from this sample as approximate estimates of the probabilities of what will happen next season." Then go on and calculate $P(team\ wins | team\ scores\ 100)$ directly from the contingency tables.

The Bayesian will object as follows: "your assumption that your sample is "representative" is unfounded. Either you have available other samples also, in which case bring them forth and prove that your current sample is representative, or you don't have other samples, so you cannot proceed as you said - your inference is unreliable". The Bayesian then would say "if we don't have other samples, the best we can do, is to accept our ignorance, then start somewhere -the prior (="before the data") and let the data modify our possibly ad hoc starting point -lead us to the posterior (="after the data")".
This means that in the Bayes' formula that appears in your question, the magnitude $P(team\ wins)$ is NOT calculated from the sample you have, but it is assigned a value a priori as the prior . And then you go to the sample to calculate $P(team\ scores\ 100 | team\ wins)$ and $P(team\ scores\ 100)$, and now you see why you have to go through Bayes formula to arrive at something that can legitimately be called the "posterior" probability $P(team\ wins | team\ scores\ 100)$: it is not calculated only from the sample at hand -if you want to use your calculations in order to say something about games that have not yet been played.

With either approach, you are now in the realm of inferential statistics.

Note: The fact that we do all these through statistical distributions and not as point probabilities (as has already been mentioned), is because we want a fuller picture of the structure of uncertainty that surrounds the future outcomes, but also, in order to calculate the uncertainty/error in our estimated probabilities.

$\endgroup$
  • $\begingroup$ Thanks for the detailed answer. Follow-up questions: (1) If I compute $P(team\ wins | team\ scores\ 100)$ directly as I suggested, would it be correctly called a 'conditional probability' rather than a 'posterior probability'? (2) Is the prior probability the ONLY difference between a frequentist's conditional probability and a Bayesian's posterior probability? That is, the Bayesian pulls the prior out of somewhere, but not from the sample? (3) Where would I get the prior if not from the sample? Would I compute it from 10 seasons of NBA games rather than a sample of one season? $\endgroup$ – stackoverflowuser2010 Sep 1 '13 at 23:55
  • $\begingroup$ (1) Yes it would be a "conditional" relative frequency, and then it could be considered as a conditional probability applying to some future events if the frequentist approach is adopted. It would never be "posterior" (2) The concept of conditional probability exists in both approaches. But in cases like yours, the Baeysian would "refuse" to consider the conditional empirical frequency as a probability that can be "projected" to unknown/future events, and would opt for the "posterior" approach instead. CONT'D $\endgroup$ – Alecos Papadopoulos Sep 2 '13 at 0:43
  • $\begingroup$ CONT'D. (3) The "prior" issue is a very big discussion (and literature) among Bayesians. E.g., look up "uninformative prior" (a reflection of total ignorance at the beginning). But of course priors can (and should) come from past data, before the current sample -like you said. If you have data for 10 seasons, you could implement 10 times the procedure: start with a "non-informative prior" and then using the data from season 1, calculate the posterior. Then use this as the prior for the next step and arrive at a new posterior by using the data from season 2, etc. $\endgroup$ – Alecos Papadopoulos Sep 2 '13 at 0:50
  • $\begingroup$ Regarding (3) and my example with 10 years of data, would it be ok to compute a prior probability for $P(team\ wins)$ directly from the winning percentage over 9 seasons, and then use the data from the 10th season to compute the likelihood $P(team\ scores\ 100 | team\ wins)$ and evidence $P(team\ scores\ 100)$, with the resulting posterior probability distribution being used to predict games in the future (after the 10th season is over)? Why not just use all 10 seasons to compute the prior, likelihood, and evidence? $\endgroup$ – stackoverflowuser2010 Sep 2 '13 at 4:55
  • $\begingroup$ Yes, you could calculate the prior as the winning percentage from the 9 periods. But in this way you would give "equal weight" to team's performance 10 years back with the team's performance last year. Is the team performance relatively stable over the years? If it is not, then such averaging over long time periods may weaken the predictive accuracy of your estimate -in which case it is better to go Bayesian, and step by step. CONTD $\endgroup$ – Alecos Papadopoulos Sep 2 '13 at 8:48
0
$\begingroup$

Your way is a natural way to compute the posterior probability, but in most Bayesian applications you are dealing with densities. As such, you want a posterior density. You don't know $f(\theta|x)$ generally, but you know $f(x|\theta)$ and $f(\theta)$ (the prior distribution). That way you can compute the posterior density of your parameter using Bayes' theorem. Also note that in many Bayesian textbooks, the notation $p(\theta|x)$ is used, but $p(\cdot)$ in this context is a density function and not a point probability.

EDIT: By density I mean distribution, yes, like the Gaussian distribution. Consider your example, but a bit extended. Let $X$ be the number of victories. What one is interested in in this case is the success probability, the parameter $\theta$. This is because the distribution of $X$ can be said to be Binomial($n, \theta$). This is your $f(x|\theta)$ -- the density function of the distribution of $X$ conditioned on the parameter. By choosing a prior distribution for $\theta$, i.e. $f(\theta)$, you can compute the posterior distribution.

I think your problem is that you mix things up. You are not computing a posterior probability as we usually speak of it. You are computing a conditional probability. Try to have a look at a Bayesian textbook if you have access to one. For example, The Bayesian Choice is available online through Springer and deals more with the computational parts of Bayesian analysis than the mathematical aspects, if I recall correctly.

$\endgroup$
  • $\begingroup$ Can you explain the difference between probability and density? By "density", do you mean "distribution", like a Gaussian distribution? By the way, I'm a software programmer, not a mathematician. Also, how can you safely say that "You don't know $f(\theta|x)$ generally, but you know $f(x|\theta)$ ..."? In my example with the logs of all NBA games, I would be able to compute $f(\theta|x)$, right? $\endgroup$ – stackoverflowuser2010 Aug 26 '13 at 0:17
  • $\begingroup$ @stackoverflowuser2010 I edited my post. Hope it clears things up a little. $\endgroup$ – hejseb Aug 26 '13 at 7:42
  • $\begingroup$ Actually, Bayes theorem applies both to continuous probabilities (probability density functions) and discrete probabilities (probability mass functions), so I don't see how this distinction is relevant to the question at hand. To give an example of a continuous variable model where the conditional probabilities are estimated directly, see [logistic regression][1]. The issue is really 'What you know' (or assume): either you know the form of $p(x|\theta)$ and $p(\theta)$, or you know the form of $p(\theta|x)$. [1]: en.wikipedia.org/wiki/Logistic_regression $\endgroup$ – John von N. Aug 27 '13 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.