Let the model distribution (likelihood) be exponential, i.e. $$ p(x \mid \lambda) := \text{Exp}(\lambda) := \lambda e^{-\lambda x} $$ and the prior distribution be gamma (shape-rate-parametrization), i.e. $$ p(\lambda \mid \alpha, \beta) := \text{Gamma}(\alpha,\beta) := \frac{\beta^{\alpha}}{\Gamma{(\alpha)}} \lambda^{\alpha-1}\exp{(-\beta\lambda)} $$ We were now asked to find the posterior distriubtion $$ p(\lambda \mid X,\alpha,\beta) = p(X \mid \lambda)p(\lambda \mid\alpha,\beta) = \text{Exp}(\lambda) \text{Gamma}(\alpha,\beta) $$ Now assuming $X = (x_k)_{k = 1}^{n}$ and i.i.d conditions, I calculated $$\text{Exp}(\lambda) = \prod_{k = 1}^{n} \lambda e^{-\lambda x_k} = \lambda^n e^{-\lambda \bar{X}},\quad \text{where } \overline{X} := \sum_{k = 1}^{n} x_k$$ and thus \begin{align} p(\lambda \mid X,\alpha,\beta) & = \lambda^n \exp\left(-\lambda \bar{X}\right) \frac{\beta^{\alpha}}{\Gamma(\alpha)} \lambda^{\alpha - 1} \exp(-\beta \lambda) \\ & = \lambda^{\alpha + n - 1} \frac{\beta^{\alpha}}{\Gamma(\alpha)} \exp\left(-\lambda \left(\beta + \overline{X}\right)\right) \end{align} and noticed $p(\lambda \mid X, \alpha, \beta) = \text{Gamma}(\alpha + n, \beta + \overline{X})$.

Until now, everything's fine.

How we were asked to find the marginal likelikhood $$ p(X \mid \alpha,\beta) = \int p(X \mid \lambda) p(\lambda \mid \alpha,\beta) \ d\lambda, $$ which I calculated to be (I used the substitution $u = \lambda(\beta + \bar{X})$) \begin{align} \int_{0}^{\infty} \lambda^{\alpha + n - 1} \frac{\beta^{\alpha}}{\Gamma(\alpha)} \exp\left(-\lambda \left(\beta + \overline{X}\right)\right) \ d\lambda & = \frac{\beta^{\alpha}}{\Gamma(\alpha)} \int_{0}^{\infty} \lambda^{\alpha + n - 1} \exp\left(-\lambda \left(\beta \overline{X}\right)\right) \ d\lambda \\ & = \frac{\beta^{\alpha}}{\Gamma(\alpha) (\beta + \overline{X})^{\alpha + n}} \int_{0}^{\infty} \lambda^{\alpha + n - 1} \exp\left(-u\right) \ d u \\ & = \frac{\beta^{\alpha} \cdot \Gamma(\alpha + n)}{\Gamma(\alpha) (\beta + \overline{X})^{\alpha + n}}. \end{align} Is this correct? Furthermore, can I find $\widehat{\alpha}$, $\widehat{\beta}$ such that $p(X|\alpha,\beta) = \text{Gamma}(\widehat{\alpha}, \widehat{\beta})$?

A similar problem arises with the predictive distribution $$ p(x \mid X,\alpha,\beta) = \int p(x \mid \lambda) p(\lambda \mid X,\alpha,\beta) d\lambda, $$ which I calculated to be (similar substitution as above) $$ \frac{\beta^{\alpha} \cdot \Gamma(\alpha + n)}{\Gamma(\alpha) (\beta + x + \overline{X})^{\alpha + n}}. $$ Is this correct? What are my $\widehat{\alpha}$ and $\widehat{\beta}$ now?


Don't make this so hard for yourself. Simply compute the kernels rather than explicitly integrating. I will change your notation because $\bar X$ is typically used for the sample mean $$\bar X = \frac{1}{n} \sum_{i=1}^n X_i$$ rather than the sample total. Then $$p(\lambda \mid X, \alpha, \beta) \propto \lambda^n e^{-\lambda n \bar X} \lambda^{\alpha - 1} e^{-\beta \lambda} = \lambda^{n + \alpha - 1} e^{-(n\bar X + \beta)\lambda}$$ which is the kernel of a gamma density with posterior shape hyperparameter $\alpha^* = n + \alpha$ and rate hyperparameter $\beta^* = n \bar X + \beta$, which agrees with your computation (keeping in mind my $\bar X$ differs from yours by a factor of $1/n$). In performing the computation, we discarded any factors that were not functions of $\lambda$. You inadvertently did this by ignoring the fact that your computation resulted in a posterior likelihood which does not integrate to unity; i.e., your expression for $p(\lambda \mid X, \alpha, \beta)$ is not a proper density since the constant terms with respect to $\lambda$ are not the required normalizing factors for a gamma density with shape $\alpha^*$ and rate $\beta^*$.

Next, if we know that the original gamma prior has a normalizing factor of $\beta^\alpha/\Gamma(\alpha)$ since $$p(\lambda, \mid \alpha, \beta) = \frac{\beta^\alpha}{\Gamma(\alpha)} K(\lambda \mid \alpha, \beta)$$ where $K$ is the kernel, and the posterior is gamma with hyperparameters $\alpha^*$, $\beta^*$, it immediately follows that the marginal likelihood is the ratio of the prior normalizing factor divided by the posterior normalizing factor; i.e., $$p(X \mid \alpha, \beta) = \frac{\beta^\alpha/\Gamma(\alpha)}{(\beta^*)^{\alpha^*}/\Gamma(\alpha^*)} = \frac{\beta^\alpha \Gamma(n+\alpha)}{(n\bar X + \beta)^{n + \alpha} \Gamma(\alpha)},$$ because of Bayes' rule: $$p(\lambda \mid X, \alpha, \beta) = \frac{p(X \mid \lambda)p(\lambda \mid \alpha,\beta)}{p(X \mid \alpha, \beta)}.$$ No integration is required. It is important to note that $p(X \mid \alpha, \beta)$ is multivariate with respect to the sample $X = (X_1, \ldots, X_n)$, thus is not itself gamma distributed.

For the posterior predictive distribution, we apply the same principles as described above. First, by Bayes' rule, $$p(x \mid X , \alpha, \beta) = \frac{p(x \mid \lambda) p(\lambda \mid X, \alpha, \beta)}{p(\lambda \mid X, x, \alpha, \beta)}$$ where the denominator is the posterior given the sample $X$ and the new observation $x$, and the numerator is the likelihood; thus the RHS is again the ratio of normalizing factors, but this time the normalizing factors correspond to the posterior density in the numerator, and the posterior plus new observation in the denominator: $$p(x \mid X, \alpha, \beta) = \frac{(\beta^*)^{\alpha^*}/\Gamma(\alpha^*)}{(\beta')^{\alpha'}/\Gamma(\alpha')},$$ where $$\alpha' = \alpha^* + 1 = n+\alpha + 1,$$ and $$\beta' = \beta^* + x = n\bar X + \beta + x.$$ Note that the posterior predictive is a univariate density in the new observation $x$, hence it is instructive to consider its kernel: $$p(x \mid X, \alpha, \beta) \propto \frac{1}{(\beta')^{\alpha'}} = (n \bar X + \beta + x)^{-\alpha'}$$ which is proportional to a Pareto (Type II) density with minimum value parameter $0$, scale parameter $\beta^* = n \bar X + \beta$ and shape parameter $\alpha^* = n + \alpha$.

  • $\begingroup$ Why do you call the $\alpha^*$ and $\beta^*$ hyperparameters and not just parameters? And how do I know that what you have computed is the "kernel of a gamma density"? $\endgroup$ – Ramanujan Feb 17 '20 at 19:36
  • $\begingroup$ Also @"You inadvertently did this by ignoring the fact that your computation resulted in a posterior likelihood which does not integrate to unity". Does posterior distribution always have to integrate to unity? $\endgroup$ – Ramanujan Feb 17 '20 at 19:38
  • $\begingroup$ Is a kernel of a distribution the distribution divided by its normalisation constant? $\endgroup$ – Ramanujan Feb 17 '20 at 19:59
  • $\begingroup$ @ViktorGlombik (1) This is a stylistic choice to emphasize that $\lambda$ is a Bayesian parameter to model the observed data $X$, and in turn $\alpha$ and $\beta$ (as well as their posterior values $\alpha^*, \beta^*$), are hyperparameters that parametrize the model for $\lambda$. The kernel is simply the component of the likelihood or density that does not depend on the variable(s) that comprise its support. Please refer to en.wikipedia.org/wiki/… for more information. $\endgroup$ – heropup Feb 17 '20 at 21:04
  • $\begingroup$ @ViktorGlombik (2) A function that is called a density for a random variable $X$ must integrate to unity. Your posterior likelihood $$p(\lambda \mid X, \alpha, \beta) = \lambda^{\alpha + n - 1} \frac{\beta^\alpha}{\Gamma(\alpha)} \exp\left(-\lambda \left(\beta + \bar X\right)\right)$$ is not in fact a gamma density because the normalization factor is incorrect. It should be $$p(\lambda \mid X, \alpha, \beta) = \lambda^{\alpha + n - 1} \frac{(\beta + \bar X)^{(\alpha+n)}}{\Gamma(\alpha+n)} \exp\left(-\lambda \left(\beta + \bar X\right)\right).$$ $\endgroup$ – heropup Feb 17 '20 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.