0
$\begingroup$

Let $Y$ be a random variable with binomial distribution of parameters $n$ and $p$. Suppose $p$ behaves as a random variable $\Pi$, $\Pi \sim \mathrm{Beta}(\alpha_1, \alpha_2)$. Obtain the conditional density of $\Pi$ given $Y = y$.

Thoughts:

This is a weird problem, I find. You have a random variable inside a random variable and you're asked about the conditional density of the "inner" random varible given a fixed value of the "outer" random variable.

Attempt:

By the definition of conditional probability,

$$ f_{\Pi |Y}(p|y) = \frac{f_{\Pi, Y}(p, y)}{P_Y(y)}$$

In the denominator, I substituted $P_Y(y) = y$ because of the condition $Y = y$. In the numerator, I just multiplied the binomial pmf by the Beta density

$$ f_{\Pi |Y}(p|y) = \frac{\binom{n}{y}p^y(1 -p)^{n-y}y^{\alpha_1 -1}(1 - y)^{\alpha_2 - 1}}{\frac{y\Gamma(\alpha_1)\Gamma(\alpha_2)}{\Gamma(\alpha_1 + \alpha_2)}} $$

Now, of course you can express $\binom{n}{y}$ as Gamma functions because $n! = \Gamma (n-1)$. Doing that, then some simplification, you get

$$ f_{\Pi |Y}(p|y) = \frac{\Gamma(n -1) \Gamma(\alpha_1 + \alpha_2)}{\Gamma(y -1) \Gamma(n - y - 1) \Gamma(\alpha_1) \Gamma(\alpha_2)} p^y(1-p)^{n-y}y^{\alpha_1 - 2}(1 - y)^{\alpha_2 - 1} $$


Official answer $$ f_{\Pi |Y}(p|y) = \frac{\Gamma(\alpha_1 + \alpha_2 + n)p^{\alpha_1 + y - 1}(1 - p)^{\alpha_2 + n - y - 1}}{\Gamma(\alpha_1 + y)\Gamma(\alpha_2 + n -y)} $$

Support is $p \in (0, 1)$

$\endgroup$
1
$\begingroup$

This hierarchical model is well-known in Bayesian probability and is sometimes described as the binomial likelihood with beta conjugate prior.

Here, it is in fact the conditional distribution $$Y \mid \Pi \sim \operatorname{Binomial}(n, \Pi).$$ The marginal or unconditional distribution $Y$ is not necessarily binomial. By Bayes' theorem, $$f(\Pi = p \mid Y = y) = \frac{\Pr[Y = y \mid \Pi = p]f_\Pi(p)}{\Pr[Y = y]}.$$ The expression on the LHS is the posterior density of $\Pi$ given the observed outcome $Y = y$. The numerator on the RHS contains the conditional probability of $Y = y$ given $\Pi = p$ (which is of course binomial) and the prior density of $\Pi$ at $p$. The denominator on the RHS is the marginal/unconditional probability of $Y = y$.

Note that from the information you are given, $$\Pr[Y = y \mid \Pi = p] = \binom{n}{y} p^y (1-p)^{n-y}.$$ You are also told $$\Pi \sim \operatorname{Beta}(\alpha_1, \alpha_2);$$ that is to say, $$f_\Pi(p) = \frac{\Gamma(\alpha_1 + \alpha_2)}{\Gamma(\alpha_1)\Gamma(\alpha_2)} p^{\alpha_1 - 1} (1-p)^{\alpha_2 - 1}, \quad 0 < p < 1,$$ which is the density of a beta prior with hyperparameters $\alpha_1$ and $\alpha_2$. The only thing you don't know is the marginal probability in the denominator. To get this, you would presumably integrate over the support of $\Pi$ using the law of total probability: $$\Pr[Y = y] = \int_{p = 0}^1 \Pr[Y = y \mid \Pi = p] f_\Pi(p) \, dp.$$ But this is totally unnecessary as we will now see. Note that the marginal probability is not a function of $p$. So, the posterior density, being a function of $p$ for a fixed $y$, is proportional to the numerator on the RHS; i.e., $$f(\Pi = p \mid Y = y) \propto \Pr[Y = y \mid \Pi = p] f_\Pi(p).$$ To this end, any multiplicative factor on the RHS that is not a function of $p$ can be omitted from our calculation, even those that contain $y$, $\alpha_1$, or $\alpha_2$. All that we care about are factors that are functions of $p$. So we get $$f(\Pi = p \mid Y = y) \propto p^y (1-p)^{n-y} p^{\alpha_1 - 1} (1-p)^{\alpha_2 - 1} = p^{y + \alpha_1 - 1} (1-p)^{n-y + \alpha_2 - 1}.$$ This expression is what we call the kernel of the posterior density. It tells us that, up to a constant (with respect to $p$) scaling factor, the posterior density is proportional to a distribution on $p \in (0,1)$ with the same kernel. This of course is a beta distribution, but the parameters of which are not the same as the prior. Clearly, the required posterior parameters need to be $$\alpha_1^* = y + \alpha_1, \quad \alpha_2^* = n-y + \alpha_2,$$ and the posterior distribution of the random variable is $$\Pi \mid Y \sim \operatorname{Beta}(Y + \alpha_1, n - Y + \alpha_2),$$ with density $$f_{\Pi \mid Y}(p \mid y) = f(\Pi = p \mid Y = y) = \frac{\Gamma(n + \alpha_1 + \alpha_2)}{\Gamma(y + \alpha_1)\Gamma(n-y + \alpha_2)} p^{y+\alpha_1 - 1} (1-p)^{n-y+\alpha_2 - 1}, \quad 0 < p < 1.$$ This is the result claimed in the quoted solution.


To provide some concrete understanding of what we are doing here, it is illustrative to consider a numeric example. Suppose I give you a coin, which may or may not be biased; and you are interested in estimating the true probability of obtaining heads. Your experiment under a Bayesian paradigm would comprise a series of coin tosses, observing the number of heads obtained as a proportion of the total number of tosses. Suppose as your first experiment, you toss the coin $n = 9$ times. Conditional on the parameter $p$, the number of heads is $Y \mid \Pi = p \sim \operatorname{Binomial}(n = 9, \Pi = p)$. As you have no prior belief or data about the value of $p$, you choose to use a uniform prior; i.e., you suppose that any value $p \in (0,1)$ is equally plausible; thus $$\Pi \sim \operatorname{Beta}(\alpha_1 = 1, \alpha_2 = 1), \quad f_\Pi(p) = 1.$$ Now suppose you observed $Y = 7$ heads. Clearly, with this information, you wish to update your belief about how $\Pi$ is distributed, since now the data suggest that the coin is biased toward heads. The above posterior formula we derived shows you exactly how to do this: Given $Y = 7$, the posterior distribution is $\operatorname{Beta}(\alpha_1^* = 7+1, \alpha_2^* = 2+1)$ and $$f_{\Pi \mid Y}(p \mid 7) = \frac{\Gamma(11)}{\Gamma(8)\Gamma(3)} p^7 (1-p)^2 = 360 p^7 (1-p)^2, \quad 0 < p < 1.$$ If you then toss the coin another $n = 12$ times, and you get $Y = 8$ heads, the posterior distribution from all of the data you have gathered so far is now $\operatorname{Beta}(16, 7)$. Note that this is the same as if you had not done two sets of trials, but a single one with $n = 21$ and $Y = 15$ heads. This is convenient property of the fact that the posterior distribution of the parameter belongs to the same family as the prior--i.e., the beta distribution is a conjugate prior for the binomially distributed data. This allows us to update our belief about $\Pi$ after each coin toss, irrespective of the order of experiments or observations.

What does the posterior distribution tell us about $\Pi$? Well, rather than giving a point estimate, or an interval estimate as a frequentist experiment would do, a posterior distribution furnishes substantially more information: it tells us what value of the parameter is likely, and the relative likelihood of one value over another. We can calculate distributional quantities such as the mode or mean to obtain point estimates, and variance for measures of the precision/uncertainty of such an estimate. We can also create interval estimates directly via the calculation of quantiles; e.g., highest posterior density intervals, or equal-tailed intervals. Such topics are beyond the scope of this discussion but should be covered in any introductory course in mathematical inferential statistics.

$\endgroup$
  • $\begingroup$ Very enlightening, thank you! $\endgroup$ – Victor S. Sep 11 at 3:26
1
$\begingroup$

You are actually provided that $(Y\mid \Pi=p)\sim\mathcal{Bin}(n,p)$, since $Y$ is binomially distributed dependent (ie conditioned) on parameter $p$, which is the realisation of another random variable $\Pi\sim\mathcal{Beta}(\alpha_1,\alpha_2)$.

$$\begin{align}\mathsf P(Y{=}y\mid\Pi{=}p)&=\binom nyp^y(1-p)^{n-y}~\mathbf 1_{y\in[0..n]\cap\Bbb N}\\[3ex]\mathsf P(Y{=}y)&=\int_0^1\mathsf P(Y{=}y\mid \Pi{=}p)~f_\Pi(p)~\mathrm d p\\[1ex]&=\mathbf 1_{y\in[0..n]\cap\Bbb N}\int_0^1 \dfrac{\Gamma(y+n-y+1)}{\Gamma(y+1)\Gamma(n-y+1)}~~p^y~(1-p)^{n-y}\cdot\dfrac{ p^{\alpha_1-1}~(1-p)^{\alpha_2-1}~\Gamma(\alpha_1+\alpha_2)}{\Gamma(\alpha_1)\Gamma(\alpha_2)}~\mathrm d p\\[1ex]&\ddots\end{align}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.