Calculating a Bayesian posterior (Jaynes' truncated example)

I've been reading a blog post highlighting differences between frequentism and bayesianism.

It present the following model:

$$p(x~|~\theta) = \left\{ \begin{array}{lll} \exp(\theta - x) &,& x > \theta\\ 0 &,& x < \theta \end{array} \right\}$$

Where the goal is to find $p(\theta | D)$, where $D$ is the given data (observations/samples from the distribution). If you follow the link, it will take you straight to the relevant calculations.

http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/#4.-Bayesian-Credibility-Interval

I'm struggling with the following.

We take $p(\theta) = 1$.

$p(D~|~\theta) = \prod_{i=1}^N p(x~|~\theta) = Nexp(\theta - x)$

In the blog post it doesn't show how $P(D)$ is derived, but I'm assuming it's by integrating the model over all possible $\theta$ values, and multiplying by the number of observations, so I get this: $P(D) =\prod_{i=1}^N \int_0^{min(D)} p(x | \theta) d\theta$

My calculation comes out to $P(D) = Ne^{-x}(e^{min(D)} - 1)$

And my posterior distribution does not match the one in the blog $$p(\theta|D) = \frac{Ne^{(\theta-x)}}{Ne^{-x}(e^{min(D)} - 1)} = \frac{e^\theta}{e^{min(D)} - 1}$$

While the blog post's posterior is $$p(\theta~|~D) \propto \left\{ \begin{array}{lll} N\exp\left[N(\theta - \min(D))\right] &,& \theta < \min(D)\\ 0 &,& \theta > \min(D) \end{array} \right\}$$

Where am I going wrong?

No wonder you're confused. The notation being used in the blog post is terribly sloppy.

Recall from the beginning that $$D = \{x_i \}_{i=1}^N = (x_1, \ldots, x_N)$$ is the sample, from which individual observations are drawn from the likelihood $$L(x \mid \theta) = \begin{cases} \exp(\theta-x), & x \ge \theta \\ 0, & x < \theta. \end{cases}$$ I use the likelihood because although I know that this function integrates to $1$ over its support, I haven't proved it, so I am content to characterize it as a likelihood rather than a density. For our purposes, this suffices.

Then the joint likelihood of observing $D$ for a fixed $\theta$ is given by $$L(D \mid \theta) = \prod_{i=1}^N L(x_i \mid \theta) = \prod_{i=1}^N \exp(\theta - x_i).$$ Note that the author forgot to include the subscript $i$.

By Bayes' rule, we then have $$L(\theta \mid D) = \frac{L(D \mid \theta)p(\theta)}{\int_{\theta \in \Theta} L(D \mid \theta)p(\theta) \, d\theta}.$$ Note I have written the marginal likelihood of $D$ in terms of an integral to show that this is not conditioned on $\theta$; and since $p(\theta) = 1$ is our chosen (improper) prior, the LHS is a function of $\theta$ for a fixed $D$, and the RHS is a function of $D$ for a fixed $\theta$, but they are proportional to each other: $$L(\theta \mid D) \propto L(D \mid \theta).$$ We can ignore the other terms because the denominator does not depend on $\theta$--it has been integrated out.

But wait!! There is a subtlety here that must not be missed, for the denominator has the interval of integration $\theta \in \Theta$. What this mean? It means that the integral is taken over all $\theta$ such that the likelihood $L(D \mid \theta)$ is positive. Thus, the integral forces us to choose $$\theta \le \min(x_1, x_2, \ldots, x_N) = \min(D),$$ and any $\theta$ larger than that corresponds to a likelihood of zero. So what we really should have written is $$L(\theta \mid D) \propto L(D \mid \theta) \mathbb{1}(\theta \le \min(D)),$$ where we have an indicator function on the condition that $\theta$ is no larger than the smallest observation. Now here's the final insight: the posterior likelihood of $\theta$ given the data (i.e., sample) depends on the sample only through its smallest observation. Everything else is a fixed constant of proportionality. That is to say, $$L(\theta \mid D) \propto L(\theta \mid \min(D)).$$ This is because $$\exp(\theta - x_i) = e^{\theta} e^{-x_i}.$$ So we can just write $$L(\theta \mid D) \propto \mathbb{1}(\theta \le \min(D)) \prod_{i=1}^N e^{\theta} = \begin{cases}e^{N\theta}, &\theta \le \min(D) \\ 0, & \theta > \min(D). \end{cases}$$ You don't actually need to write the posterior likelihood as $N \exp(N(\theta - \min(D)))$; the $N$ is constant, and after expanding out the exponent, you'd just get $$\exp(N(\theta-\min(D))) = e^{N\theta} e^{-N\min(D)}$$ and the second factor is also constant with respect to $\theta$. This is why we've been dealing in likelihoods all throughout. If you wished to normalize this likelihood to get a proper posterior distribution of $\theta$, then it becomes clear that it is $$f(\theta \mid D) = \begin{cases} N \exp(N(\theta - \min(D))), & \theta \le \min (D) \\ 0, & \text{otherwise}, \end{cases}$$ the likelihood quite obviously belonging to a location-scale transformed exponential distribution (namely, the transformation is $\theta = (\min(D) - Y)/N$ where $Y \sim \operatorname{Exponential}(1)$.

So that's all the mathematical statistical background behind that particular part of the post. It's a lot longer than it needs to be--really, the concepts are quite simple. One of your issues is that you are integrating from $\theta = 0$ to $\theta = \min(D)$; first of all, as I have shown, you don't need to do any integration at all because the denominator, as I have explained, is only a function of $\theta$ through the condition $\theta \le \min(D)$; and second, the lower limit would have been $\theta = -\infty$ if you were to have integrated it anyway. What I want you to take away from this is that working with likelihoods does away with having to worry about proper probability densities for such simple and tractable parametric models. Normalize the posterior at the end of the computation.

• Thank you for the explanation. It's great! I'm not clear on one point though: You have $L(\theta | D) \propto \begin{cases}e^{N\theta}, &\theta \le \min(D) \\ 0, & \theta > \min(D). \end{cases}$. For a calculation (if you had actual data), where would $e^{-Nmin(D)}$ come from? Would that be part of normalizing the likelihood to become a distribution? – Andrey Sep 23 '15 at 14:43
• @Andrey Yes. The posterior likelihood $L(\theta \mid D)$ is not normalized. If you integrated $L(\theta \mid D)$ for $\theta \in (-\infty, \min(D))$, and divided $L(\theta \mid D)$ by this normalizing constant, you will get exactly $f(\theta \mid D)$. But an easier way to do see it, in my opinion, is to recognize that $e^{N\theta}$ is proportional to $Ne^{N\theta}$ which is an exponential distribution. Then the requirement $\theta \le \min(D)$ tells us this exponential distribution has been shifted, so we just need to look for the correct location and scale parameters. – heropup Sep 23 '15 at 15:46