I have been struggling on a homework question where I have to compute the posterior density of a distribution. While I can compute the posterior, I believe I made a mistake because the area under the posterior density tends to infinity.

I have spent hours double checking my work and have had several friends also come up with the same answer, so I was hoping that someone could take a look at the work and either spot the mistake or explain what is going on.

The setup is as follows. Let:

  • $y = (y_1,\ldots,y_N)$
  • $p_x(x) = \exp(-x)$
  • $p_{y_i|x} = xy_i^{x-1} = x\exp((x-1)\log{y_i})$ for $y_i \in (0,1)$ and $x>0$

Given this information, we can calculate $p_{x|y}(x|y)$ using Bayes' Rule as follows:

$$\begin{align} p_{x|y}(x|y) &\propto p_{y|x}(y|x)p_x(x) \\ &\propto \prod_{i=1}^N p_{y_i|x}(y_i|x)p_x(x)\\ &\propto \prod_{i=1}^N x\exp((x-1)\log{y_i})\exp(-x)\\ &\propto x^N\exp((x-1)\sum_{i=1}^N\log{y_i})\exp(-Nx)\\ &\propto x^N\exp(x(\sum_{i=1}^N\log{y_i}-N))\exp(\sum_{i=1}^N\log{y_i})\\ \end{align}$$

Letting $S=\sum_{i=1}^N\log{y_i}$, this results in:

$$\begin{align} p_{x|y}(x|y) = \frac{1}{Z}x^N\exp(x(S-N)\exp(-S) \end{align}$$

Where $Z$ denotes a normalization constant that can be calculated as:

$$\begin{align} Z &= \int_{x>0}p_{y|x}(y|x)p_x(x)dx \\ &= \int_{x>0}x^N\exp(x(S-N))\exp(-S)dx \\ &= \exp(-S)\frac{1}{N-S}\int_{x>0}x^N(N-S)\exp(-x(N-S))\exp(-S)dx \\ &= \exp(-S)\frac{1}{N-S}\mathbb{E}[X^N] \\ &= \exp(-S)\frac{1}{N-S}\frac{N!}{(N-S)^N}\\ &= \exp(-S)\frac{N!}{(N-S)^{N+1}}\\ \end{align}$$

Note that I have evaluated the above integral using the $N$th moment of an exponential random variable.

Putting everything together, this gives me:

$$\begin{align} p_{x|y}(x|y) &= \frac{x^N\exp(x(S-N))\exp(-S)}{\exp(-S)\frac{N!}{(N-S)^{N+1}}}\\ &= \frac{(N-S)^{N+1}}{N!} x^N\exp(x(S-N))\\ \end{align}$$

Unfortunately, this density approaches infinity as $x \rightarrow \infty$, and so it does not normalize to 1.

  • $\begingroup$ "this density approaches infinity as x→∞"... how do you arrive at that conclusion? $\endgroup$ – Glen_b Apr 24 '13 at 5:00

Your conclusion is not correct. Consider

$$ \int_0^\infty p(x|y)dx=\int_0^\infty (S-1)^2x e^{(-x(S-1))}dx=(S-1)\int_0^\infty (S-1)x e^{(-x(S-1))}dx=(S-1)E(V) $$ where $V\sim Exp\left(\frac{1}{(S-1)}\right)$. The expected value of an exponentially distributed variable is its parameter. Hence,

$$ \int_0^\infty p(x|y)dx=(S-1)E(V)=\frac{(S-1)}{(S-1)}=1 $$

and it integrates to 1, as it should.

  • $\begingroup$ Thank you for this. I just spotted an extra negative sign that snuck into my final expression. $\endgroup$ – Elements Apr 24 '13 at 5:53
  • $\begingroup$ Hmm, I used a negative sign in front of $x$ in my calculations (which you apparently didn't have), but that should not change anything. Since $y_i\in(0,1)$ the sum will be negative. Since the parameter in an exponential must be positive, you should probably define $S$ as $-\sum_{i=1}^n\log y_i$ instead of without the negative sign. And then you get a negative sign in front of the $x$ as I then used. $\endgroup$ – hejseb Apr 24 '13 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.