# Bayesian statistics to find a distribution

Suppose that a random sample $$X_1, ..., X_n$$ is taken from a distribution where $$X$$ is a random variable that follows a Poisson distribution with mean $$w$$. Furthermore, $$w$$ is modeled with a random variable $$W$$ that follows a chi-square distribution with $$r = 4$$ degrees of freedom. (i) Determine the posterior distribution of $$W$$ given that $$X_1 = x_1, X_2 = x_2, ..., X_n = x_n$$ (ii) Find the Bayesian estimator for $$W$$ that minimizes the mean-square error (iii) Before the sample was taken, Jack tells you that $$w$$ lies in $$(0, 1)$$, however, no further information is provided about which values $$w$$ would most likely take in this interval. Do you pick a new prior pdf for $$W$$? Justify your answer, stating what new prior pdf you would take if so, or if not, why you would not pick a new prior pdf.

Hint: Integration is unnecessary. The posterior distribution should be familiar if you use proportionality! You should use this posterior distribution to find the mean-square error.

My attempt:

(i) From chi-square with $$r = 4$$, the prior pdf of $$W$$ is

$$f_W(w) = \frac{we^{-w/2}}{4}$$

The pdf of a Poisson with mean $$w$$ is:

$$f(x|w) = \frac{w^{x}e^{-w}}{x!}$$

The joint pdf of $$X_1, X_2, ..., X_n$$ is:

$$F(x_1, ..., x_n|w) = \frac{w^{\sum{x_i}}e^{-nw}}{\prod{x_i!}}$$

The posterior probability density function (pdf) of $$W$$ given $$X_1 = x_1, X_2 = x_2, ..., X_n = x_n$$ is:

$$f(w|x) = \frac{F(x_1, ..., x_n|w) \cdot f_W(w)}{f_X(x)} = \frac{F(x_1, ..., x_n|w) \cdot f_W(w)}{\int_{-\infty}^{\infty} F(x_1, ..., x_n|w) \cdot f_W(w) \,dw}$$

We apply proportionality:

$$\frac{F(x_1, ..., x_n|w) \cdot f_W(w)}{\int_{-\infty}^{\infty} F(x_1, ..., x_n|w) \cdot f_W(w) \,dw} \propto F(x_1, ..., x_n|w) \cdot f_W(w) = \frac{w^{1+\sum{x_i}}e^{-nw-\frac{w}{2}}}{4\prod{x_i!}}$$

From here, there are a few possible posteriors to choose from: Chi-square, Exponential or Gamma.

By mere inspection:

If Chi-square, we have parameter $$r = 4 + 2\sum{x_i}$$.

If Exponential, we have parameter $$\theta = \frac{-2}{w(2n+1)}$$ or $$\theta = \frac{1}{w^{1+\sum{x_i}}}$$.

If Gamma, we have parameters $$\alpha = 2 + \sum{x_i}$$ and $$\theta = \frac{2}{2n+1}$$.

I am unsure which one is the correct posterior distribution as they all seem likely.

(ii) Depends on (i) but am I supposed to use the fact that $$E(W|X)$$ is the least mean square here?

(iii) Not sure.

Any assistance especially with (i) and (iii) is much appreciated.

• For (i), it is obviously not exponential since there cannot be a $w$ term in it apart from the exponential. Note also that your parameter $\theta$ must be a constant for an exponential distribution and cannot depend on the variable of interest ($w$), and furthermore, it is required that $\theta > 0$. It also can't be chi-squared because the exponent on $e$ must be equal to $-w/2$. Thus, Gamma is probably correct, but note: for Gamma, your $\theta$ needs to be $> 0$. Check your algebra and compare with the Gamma PDF. Commented May 19, 2021 at 20:47
• Are you defining an exponential distribution with parameter $\theta$ to have density function under definition 1: $f(x) = \theta e^{-\theta x}$, $x > 0$ - or definition 2: $f(x) = \dfrac{1}{\theta}e^{-x/\theta}$, $x > 0$? Commented May 19, 2021 at 20:56
• @Clarinetist Thank you for the response. The second definition. Commented May 19, 2021 at 21:01
• For (iii) you might consider a uniform$(0, 1)$ prior. The chi squared distribution is nonzero for values far from $(0, 1)$, so might not be as good of a prior as a uniform$(0, 1)$. Commented May 19, 2021 at 21:23
• @SupremePickle chi-squared $r$ distribution is the distribution of the sum of squares of $r$ i.i.d. normal$(0, 1)$ variables. So the chi-squared $r$ density is nonzero for values outside of $(0, 1)$. See the graphs here: en.wikipedia.org/wiki/Chi-square_distribution Commented May 19, 2021 at 21:37

For (i), you must be careful of the assumptions made in the definitions of the density functions.

For the exponential distribution, we have the density function $$f_{X}(x) = \dfrac{1}{\theta} e^{- x/\theta}\text{, }\qquad x > 0$$ with $$\theta > 0$$ a constant - i.e., it cannot depend on $$x$$.

For the Gamma distribution, we have the density function $$f_{X}(x) = \dfrac{1}{\Gamma(\alpha)\theta^\alpha}x^{\alpha - 1}e^{-x/\theta}\text{, }\qquad x > 0$$ with $$\alpha > 0$$ and $$\theta > 0$$ constants - again, not depending on $$x$$.

The Chi-squared distribution with $$\nu$$ degrees of freedom is just the Gamma distribution with $$\alpha = \nu/2$$ and $$\theta = 2$$.

As far as I can tell, your computation of the posterior PDF is correct: we know that $$f(w \mid x) \propto \frac{w^{1+\sum{x_i}}e^{-nw-\frac{w}{2}}}{4\prod{x_i!}}$$ but we can take this one step further: note that $$w$$ is the variable of interest, so we can clean this up to end up with $$f(w \mid x) \propto \frac{w^{1+\sum{x_i}}e^{-nw-\frac{w}{2}}}{4\prod{x_i!}} \propto w^{1+\sum{x_i}}e^{-nw-\frac{w}{2}}\text{.}$$ Now, we observe that this is indeed similar to the exponential, Gamma, and Chi-squared distributions. However, for all three of these, we must separate the variable of interest in the exponent in the numerator - i.e., this suggests $$f(w \mid x) \propto w^{1+\sum{x_i}}e^{-w\left(n+\frac{1}{2}\right)}\text{.}$$ This has a $$w$$ term in it, which would not be the case for the exponential distribution (to match the notation of the exponential PDF above, replace $$w$$ with $$x$$). It is also not Chi-squared, since the factor of the exponent of $$e$$ should be $$1/2$$. So it's probably Gamma, and we can see this by writing $$-w\left(n + \dfrac{1}{2}\right) = -\dfrac{w}{\left(n + \dfrac{1}{2}\right)^{-1}}$$ which yields $$\theta = \left(n + \dfrac{1}{2}\right)^{-1} > 0$$ and $$\alpha = 1 + \sum x_i + 1 = 2 + \sum x_i > 0$$, thus we have a Gamma PDF.

Your thought for (ii) is correct; given (i), this should be easy to find since you know the posterior distribution.

I don't have enough experience with Bayesian model fitting to answer (iii).

• This makes a lot of sense, thanks! Minor error though: you forgot to factor out the negative earlier on. Commented May 19, 2021 at 21:19
• @SupremePickle Ah, thanks! Editing now. Commented May 19, 2021 at 21:20
• Sorry but I had a follow-up question: Can't we factor out the $\frac{1}{2}$ in the exponent as follows: $-\frac{1}{2}w(2n+1)$? Commented May 20, 2021 at 0:11
• @SupremePickle Sure, but $w(2n+1)$ is not the variable of interest; you can't say it is Chi-squared on that basis. Commented May 20, 2021 at 1:55