I'm reading an introduction to Bayesian updating which includes the following example (by the way, I will add my reasoning at some parts so you guys can tell me whether I'm correct or not):

Let $\mu_{m}$ be the model that asserts $P(head) = m$. Let s be a particular sequence of observations yielding i heads and j tails. Then, for any m, $0\leq m \leq 1$:

$$P(s|\mu_{m}) = m^{l}(1-m)^{j} \tag{*}$$

Good enough. It's a situation where there are two possible outcomes (heads or tails), so a distribution like that is sensible enough. Basically, I'm asking what is the probability of having a given sequence of heads and tails ($s$) given a model with some m. This example goes on to describe the difference between a frequentist approach and a bayesian approach and says:

But now suppose that one wants to quantify one's belief that the coin is probably a regular, fair one. One can do that by assumming prior probability distribution over how likely it is that different models $\mu_{m}$ are true.

Because of some nice properties, the author chooses the following distribution:


So, if I want to know the new probability of having a fair or unfair coin (which is equivalent to asking the probability for a given model $\mu_{m}$) to be true then I have to use Bayes' theorem:

$$P(\mu_{m}|s) = \frac{P(s|\mu_{m})P(\mu_{m})}{P(s)} \tag{1}$$

Good, we plug in our prior probability $P(\mu_{m})$ and our value for $P(s|\mu_{m})$ in $(1)$, which gives $\displaystyle \frac{6m^{i+1}(1-m)^{j+1}}{P(s)}$. We don't need the denominator to calculate the maximum value $m$ for which $(1)$ is a maximum probability, that is, for that value of $m$ the model is more likely to be true given the data $s$. It turns out that, in this case, that value is $\frac{3}{4}$ (assuming 8 heads and 2 tails).

Now, we never calculated $P(s)$ but the author says that this is a marginal probability, so we need to do this:

$$P(s) = \int_{0}^{1}P(s|\mu_{m})P(\mu_{m})dm \tag{2}$$

I sort of understand why we do this. We want the prior probability of $s$ but we can't ask the probability of having $s$ (a given series of heads and tails) without any assumption. However, assuming a given model $(*)$, we can ask that for some $m$ and of course, not only that, we have to consider every m. The fact that it's a weighted sum is reasonable but I don't have a good explanation for it. Obviously, that's the way marginal probabilities work but still, it's a bit uncomfortable just to assume that's the way it is. I'd like to know why. Another question: Is the denominator in Bayes's equation always a marginal (prior) probability?

The solution for $(2)$ is $P(s)=\displaystyle \frac{6(i+1)!(j+1)!}{(1+j+3)!}$.

Do I need to use that $P(s)$ in $(1)$ to get a probability of a particular model $\mu_{m}$ being true given a sequence of heads/tails $s$? If I choose to use $m=1/2$, am I disregarding the maximum value $m=3/4$ in favor of the assumption of having a fair coin?

Thanks a lot


Maybe a change of notation will help here. Consider $X_{1},\ldots,X_{n}$ a sequence of coin flips and $\Theta$ a random variable such that:

$$P(X_{1}=x_{1},\ldots,X_{n}=x_{n}|\Theta=\theta) = \theta^{n\bar{x}}(1-\theta)^{n-n\bar{x}}$$

This is exactly the same model you described. You are also defining how to simulate from $X_{1},\ldots,X_{n}$. You can simulate from $\Theta$ and then, $X_{1},\ldots,X_{n}$ as coin flips from a coin with probability of heads equal to the outcome of $\Theta$. In other words, you are making assumptions about $X_{1},\ldots,X_{n}$ behave even when the value of $\Theta$ is unknown. Is this reasonable?

Maybe de Finetti's Representation Theorem might help you. Consider a sequence $(X_{n})_{n \in N} \in \{0,1\}^{N}$ such that, for any $n$ and any $\pi$ permutation of the indexes, $P(X_{1}=x_{1},\ldots,X_{n}=x_{n}) = P(X_{1}=x_{\pi(1)},\ldots,X_{n}=x_{\pi(n)})$. The theorem states that then there exists a $\Theta$ such that:

$$P(X_{1}=x_{1},\ldots,X_{n}=x_{n}|\Theta=\theta) = \theta^{n\bar{x}}(1-\theta)^{n-n\bar{x}}$$

If this assumption is reasonble than by the law of total probability so is the way you computed the marginal of $X$.

Next, observe that the prior used by the author assumes that the probability that the coin is fair is $0$. Hence, no matter what happens, you will continue to believe that the probability that the coin is fair is $0$. If that is not your prior opinion, you should not use this beta prior (even though its mathematical properties are convenient).

Also, Bayes' Theorem states that:

$$P(X=x|D=d) = \frac{P(X=x)P(D=d|X=x)}{P(D=d)}$$

Let $D$ represent your data. You usually know $P(D=d|X=x)$, which is the probability of obtaining your data under some model. The unconditional probability (also called predictive probability of D) might be more difficult to elicit. In this case, you can use:

$$P(D=d) = \int{P(D=d,X=x)dx} = \int{P(D=d|X=x)f(x)dx}$$

Nevertheless, nothing says that you have to specify the model in this order. For example, instead of specifying $P(X=x)$ and $P(D=d|X=x)$ yo might want to specify $P(X=x,D=d)$.

  • $\begingroup$ Thanks for your answer. Are you saying that the distribution $P(\mu_{m})$ sets the probability of a fair coin as 0 or what do you mean by this "If this assumption is reasonble than by the law of total probability..." As for you last equation $P(D=d)...$, are you sure about that? It seems there is something missing there. $\endgroup$ – Robert Smith Jun 30 '12 at 23:43
  • $\begingroup$ If the assumption in de Finetti's Theorem is reasonable, then $P(s)$ has to be of the form you described in your post. Also, $P(\Theta = 0.5) = 0$ is a consequence of the prior adopted by the author. The notation in $P(D=d)$ is goofy, but I think the general idea is right. $\endgroup$ – madprob Jul 1 '12 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.