The problem:

Given a known quantity $x$, distributed with known distribution $π(x)$ ~ $N(0,σ^2)$, I'm looking for the distribution of the estimator of $x$, $\hat{x}$ distributed with $p(\hat{x}\mid x)$ by minimizing

$F = \langle (x-\hat{x})^2\rangle _{x,\hat{x}}+βI(x,\hat{x})$ with respect to $p(\hat{x}\mid x)$.

where $I(x,\hat{x})$ is the mutual information between the two quantities.

My attempt:

$\langle (x-\hat{x})^2\rangle _{x,\hat{x}} = \mathbb E_x[\mathbb E_{\hat{x}|x}[(x-\hat{x})^2\mid x]] = \int_{-\infty}^\infty (x-\hat{x})^2p(\hat{x}\mid x) \pi(x)dx$

$I(x,\hat{x}) = \mathbb E[\log \frac{p(\hat{x}|x)}{p(\hat{x})}] = \int p(\hat{x}\mid x)\log p(\hat{x}\mid x)-(\int p(\hat{x}\mid x')\pi(x')dx')\log(\int p(\hat{x}\mid x'')\pi(x'')dx'')dx$

I think I made an error in $I(x,\hat{x})$...

But taking a functional derivative of $F$ w.r.t. $p(\hat{x}\mid x)$ gives me:

$\int_{-\infty}^\infty (x-\hat{x})^2\pi(x)dx + β\int \{\log p(\hat{x}\mid x)-\log[\int p(\hat{x}\mid x')\pi(x')dx']\} $

I would set this to 0 to find $p(\hat{x}\mid x)$, but I'm not certain this is right.


1 Answer 1


Your problem seems unconventional and of no clear utility. Are you sure you have posed it correctly? Assuming you have, how about this approach:

It is well-known (you can show it) that the optimal $\hat{x}$ that minimizes the mean squared error (MSE) $\mathbb{E}((x-\hat{x})^2) $ is $\hat{x}=\mathbb{E}(x)$. Since, by default, $\mathbb{E}(x)=0$, it follows that the MSE-minimizing (conditional) distribution for $\hat{x}$ is the degenerate distribution $p(\hat{x}|x)=\delta(\hat{x})$, where $\delta(\cdot)$ is the Dirac delta function. Note that the MSE-minimizing $\hat{x}$ is independent of $x$.

Now, consider the mutual information term. It is well-known (you can show it) that $I(x;\hat{x})\geq 0$ with the lower bound achieved for any $\hat{x}$ that is indepedent of $x$. Clearly, one such choice that achieves the bound is a $\hat{x}$ disributed as the MSE-minimizing distribution $p(\hat{x}|x)=\delta(\hat{x})$.

Since $p(\hat{x}|x)=\delta(\hat{x})$ is a minimizer for both factors of your cost function, it is also a minimizer for the cost function as well.

  • $\begingroup$ I understand what you've written but perhaps I haven't posed the problem properly. I'll try to think about what I actually want to say and form my question better. Thank you for the help though! $\endgroup$
    – JSL
    Jul 30, 2018 at 15:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .