# How to find the conditional distribution of an estimator given a prior

### The problem:

Given a known quantity $$x$$, distributed with known distribution $$π(x)$$ ~ $$N(0,σ^2)$$, I'm looking for the distribution of the estimator of $$x$$, $$\hat{x}$$ distributed with $$p(\hat{x}\mid x)$$ by minimizing

$$F = \langle (x-\hat{x})^2\rangle _{x,\hat{x}}+βI(x,\hat{x})$$ with respect to $$p(\hat{x}\mid x)$$.

where $$I(x,\hat{x})$$ is the mutual information between the two quantities.

### My attempt:

$$\langle (x-\hat{x})^2\rangle _{x,\hat{x}} = \mathbb E_x[\mathbb E_{\hat{x}|x}[(x-\hat{x})^2\mid x]] = \int_{-\infty}^\infty (x-\hat{x})^2p(\hat{x}\mid x) \pi(x)dx$$

$$I(x,\hat{x}) = \mathbb E[\log \frac{p(\hat{x}|x)}{p(\hat{x})}] = \int p(\hat{x}\mid x)\log p(\hat{x}\mid x)-(\int p(\hat{x}\mid x')\pi(x')dx')\log(\int p(\hat{x}\mid x'')\pi(x'')dx'')dx$$

I think I made an error in $$I(x,\hat{x})$$...

But taking a functional derivative of $$F$$ w.r.t. $$p(\hat{x}\mid x)$$ gives me:

$$\int_{-\infty}^\infty (x-\hat{x})^2\pi(x)dx + β\int \{\log p(\hat{x}\mid x)-\log[\int p(\hat{x}\mid x')\pi(x')dx']\}$$

I would set this to 0 to find $$p(\hat{x}\mid x)$$, but I'm not certain this is right.

It is well-known (you can show it) that the optimal $\hat{x}$ that minimizes the mean squared error (MSE) $\mathbb{E}((x-\hat{x})^2)$ is $\hat{x}=\mathbb{E}(x)$. Since, by default, $\mathbb{E}(x)=0$, it follows that the MSE-minimizing (conditional) distribution for $\hat{x}$ is the degenerate distribution $p(\hat{x}|x)=\delta(\hat{x})$, where $\delta(\cdot)$ is the Dirac delta function. Note that the MSE-minimizing $\hat{x}$ is independent of $x$.
Now, consider the mutual information term. It is well-known (you can show it) that $I(x;\hat{x})\geq 0$ with the lower bound achieved for any $\hat{x}$ that is indepedent of $x$. Clearly, one such choice that achieves the bound is a $\hat{x}$ disributed as the MSE-minimizing distribution $p(\hat{x}|x)=\delta(\hat{x})$.
Since $p(\hat{x}|x)=\delta(\hat{x})$ is a minimizer for both factors of your cost function, it is also a minimizer for the cost function as well.