2
$\begingroup$

It is given that $\tilde{x} = \theta + \varepsilon$, where $\theta \sim N(y, 1/a)$ and $\varepsilon \sim N(0, 1/b)$.

Conditional on that we observe a realization $\tilde{x} = x$, what will be the conditional distribution of $\theta$?

It seems the correct answer would be a normal distribution with a mean of $\frac{ay + bx}{a+b}$ and a variance of $\frac{1}{a+b}$. But I am not sure how to get there.

I thought that conditional on $\tilde{x} = x$, one can treat $x$ as a constant so that $\theta|_{\tilde{x}=x} \sim N(x,1/b)$. But apparently I got something wrong.

Can anyone help? Thanks a lot!

$\endgroup$
1
  • $\begingroup$ You haven't told us the joint distribution of $\theta$ and $\varepsilon,$ but only their marginal distributions. Did you intend them to be independent? If so, you need to say so. $\endgroup$ Commented Jan 26, 2017 at 0:28

2 Answers 2

1
$\begingroup$

I'm going to assume you intended $\theta$ and $\varepsilon$ to be independent. Omission of that information is a frequent mistake.

Your proposal that the conditional expected value of $\theta$ given $\tilde x = x$ is $x$ would be right if $\tilde x$ and $\varepsilon$ were independent. But if $\theta$ and and $\varepsilon$ are independent, then the correlation between $\tilde x = \theta+\varepsilon$ and $\theta$ is positive, not zero.

Consider the case where the variance of $\theta$ is small and the variance of $\varepsilon$ is large: in that case, $\tilde x$ being a long way from $y$ would more likely result from variation of $\tilde x$ than from $\theta$ being far from $y$, so the conditional expected value of $\theta$ would certainly not be $\tilde x$.

One way to look at conditional distributions in this situation is via Bayes's theorem, which says the posterior distribution is proportional to the product of the prior distribution and the likelihood function. The prior distribution of $\theta$ is proportional to $$ \sqrt a \exp\left( \frac{-1}2 a^2 (t-y)^2 \right) \,dt. $$ The likelihood function given the observation that $\tilde x = x$ is $$ L(t) = \sqrt b \exp\left( \frac{-1}2 b^2 (x - t)^2 \right). $$ Notice that in the first line above you see $dt$, indicating that this is a measure on the $t$-axis, and no $dt$ appears in the second line; this is not a measure but a pointwise defined function. Multiplying a measure by a pointwise defined function yields a measure, so you expect to see $dt$ again.

To multiply these, we add the exponents: \begin{align} & a(t-y)^2 + b(x-t)^2 \\[10pt] = {} & (a+b) t^2 - 2(ya+xb)t + \text{constant} \\ & \quad \text{(and in this case “constant" means not depending on $t$)} \\[10pt] = {} & (a+b) \left( t^2 - 2 \frac{ya+xb}{a+b} t \right) + \text{a different constant} \\[10pt] = {} & (a+b) \left( t - \frac{ya+xb}{a+b} \right)^2 + \text{yet another constant}. \end{align} We get the posterior distribution $$ \text{constant} \times \exp\left( (a+b) \left( t - \frac{ya+xb}{a+b} \right)^2 \right) \, dt. $$ Hence the posterior distribution is a normal distribution with variance $1/(a+b)$ and expected value is $(ya+xb)/(a+b).$

As a way to check this, find $$ \operatorname{cov}\left( \tilde x, \theta - \frac{ay+b\tilde x}{a+b} \right), $$ You should get $0$, so they're uncorrelated, and in a multivariate normal setting that means they're independent. If two random variables are independent, then the conditional expected value of the first one given the second, is the first one, so we have $$ \operatorname{E}\left( \theta - \frac{ay+b\tilde x}{a+b} \mid \tilde x \right) = 0, $$ and that implies $$ \operatorname{E}\left( \theta \mid \tilde x \right) - \operatorname{E}\left( \frac{ay+b\tilde x}{a+b} \mid \tilde x \right) = 0. $$ Since the second conditional expected value is determined by $\tilde x$, we have $$ \operatorname{E}( \theta \mid \tilde x) - \frac{ay+b\tilde x}{a+b} = 0. $$

$\endgroup$
2
  • $\begingroup$ Thank you so much! I do realize why I was wrong, now, trying to follow your proof. $\endgroup$
    – Hyperplane
    Commented Jan 26, 2017 at 2:14
  • $\begingroup$ BTW, I confirm that my answer had an error (see the comment) and this answer by MH is correct. $\endgroup$ Commented Jan 26, 2017 at 16:35
0
$\begingroup$

The joint distribution is $$ f(\theta,\epsilon) = K(a,b) e^{-\frac12 \left( (y-\theta)^2a + \epsilon^2 b \right)} $$ where $K(a,b)$ is some normalization factor like $\frac{\sqrt[4]{ab}}{2\pi}$ that we need not care about.

Using $\theta = \tilde{x} - \epsilon$ we can move over to use $\tilde{x}$ instead of $\theta$ as one variate:$$ f(\tilde{x},\epsilon) = K(a,b) e^{-\frac12 \left( (y-\tilde{x}+\epsilon)^2a + \epsilon^2 b \right)}$$Then we find the marginal distribution of $\epsilon$ by setting $\tilde{x} = x$: $$f(\epsilon\mid\tilde{x}=x) = K(a,b) e^{-\frac12 \left( (y-x+\epsilon)^2a + \epsilon^2 b \right)}$$ Anticipating that this will be a normal distribution, we can work with just the quantity in the exponent, and complete the square: $$ -\frac12 \left( (y-x+\epsilon)^2a + \epsilon^2 b \right) = -\frac12 \left( \epsilon^2(a+b) -2\epsilon(a^2x-ay) +a(y-x)^2 \right)\\ = -\frac12 (a+b) \left( \epsilon^2-2\epsilon\frac{ax-ay}{a+b} + \frac{a(y-x)^2}{a+b} \right)\\ = -\frac12 (a+b) \left( \epsilon-\frac{ax-ay}{a+b}\right)^2 - M(a,b,x,y) $$ where $e^{-\frac12 (a+b) M(a,b,x,y)}$ is again a normalization factor that we need not be concerned about (although the careful calculator will keep track to make sure the integral of the marginal distribution indeed works out to one).

We now recognize that $$ f(\epsilon) \propto e^{-\frac12 (a+b) \left( \epsilon-\frac{ax-ay}{a+b}\right)^2} $$ is a normal distribution $$N\left( \frac{ax-ay}{a+b},\frac1{\sqrt{a+b}} \right)$$

This is different than the answer you said "seems correct." You can tell that this answer is wright because when $a$ and $b$ are very large, the marginal distribution of
$\epsilon$ must be strongly peaked about $x-y$.

$\endgroup$
2
  • $\begingroup$ Your conditional expected value has a minus sign where a plus sign should appear. It implies that as $y$, the prior expected value, gets smaller, then the conditional expected value gets bigger. Think about what that implies. $\endgroup$ Commented Jan 26, 2017 at 1:59
  • $\begingroup$ @Michael Hardy: Yes, you are right. I was finding the marginal distribution of $\epsilon$ (as stated in the phrase "Then we find the marginal distribution of $\epsilon$," and at the bottom, where $f(\epsilon)$ is presented). The problem asked for the marginal distribution of $\theta$. $\endgroup$ Commented Jan 26, 2017 at 16:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .