I am trying to understand Bayesian estimation and I come across this line in my lecture notes:

 θ(Bayesian) = E_θ|x[θ] =  E[π(θ|x)]

So it's meant to reader that the Bayesian estimator is the Conditional Expectation of the sample (x's) which equals the expectation of the posterior (3'rd expression). I understand the derivation up to this point but I dont see how the conditional expectation of theta:

So the middle expression "E_θ|x[θ]" (should look like E underscore θ|x of θ) is:

Integral [θ · π(θ|x) dθ]

and somehow that equals the expectation of the Posterior i.e.


Please help! I am hoping it is a simple answer cause there is no explanation between these steps.

  • $\begingroup$ I've tried thinking about it as E_Y|X (y) = E [ f_Y|X (y|x) ] if that helps but I haven't been able to find a solution, probably because I don't know how to deal with the RHS expectation of a conditional density.... $\endgroup$ – Mauro Augusto May 20 '16 at 13:36
  • $\begingroup$ I guess its a matter of notation. Let $\pi(\theta |x)$ be the posterior, then the posterior mean (which is the Bayesian estimator), denoted as $\mathbb{E}[\pi(\theta |x)]$, is given by $$\int \theta \ \pi(\theta | x)d\theta.$$ The above is simply the conditional expectation of $\theta$ w.r.t the observations $x$ and this is exactly the posterior mean. $\endgroup$ – Cavents May 20 '16 at 14:30
  • $\begingroup$ I get that the expression you wrote out is the conditional expectation, thats what a bunch of the derivation leads up to. But how does that equal to the Expectation of the posterior. I mean thats what we are trying to prove isnt it? It seems to me like you are using the result of the proof in the proof. $\endgroup$ – Mauro Augusto May 20 '16 at 15:25
  • $\begingroup$ Hi -- welcome to math.SE! Here's a reference and tutorial for typesetting math on this site. $\endgroup$ – joriki May 20 '16 at 16:43
  • $\begingroup$ @MauroAugusto: because $\pi(\theta|x)$ is the posterior density which is a function of $\theta$. Therefore, the integral is exactly the definition of the mean of the posterior (.... recall the definition of the expected value of a random variable). $\endgroup$ – Cavents May 20 '16 at 17:24

At the begin, you should make sure what is the Bayesian estimator. In chapter 11 of [1], Bayesian estimator is defined as an estimator which minimizes the Bayesian mean square error (Bayesian MSE). Let $\boldsymbol{x}$ be an observed signal, $\theta$ is estimated signal and $\hat{\theta}$ is estimator of $\theta$. With those knowledge, we have \begin{align} \text{Bmse}\ (\hat{\theta}) &=\int (\hat{\theta}-\theta)^2p(\boldsymbol{x},\theta)\text{d}\boldsymbol{x}\text{d}\theta\\ &=\int \left[{\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}x}\right]p(\theta)\text{d}\theta \end{align} Thanks to the non-negativity of $p(\theta)$, the minimum of Bayesian MSE can be touched via minimizing inner integral. To this end, we take partial derivative of inner integral w.r.t $\hat{\theta}$ \begin{align} \frac{\partial }{\partial \hat{\theta}}\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}\boldsymbol{x}=-2\int \theta p(\theta|\boldsymbol{x})\text{d}\theta+2\hat{\theta}\int p(\theta|\boldsymbol{x})\text{d}\theta \end{align} Let it be 0 yields \begin{align} \hat{\theta}=\int \theta p(\theta|\boldsymbol{x})\text{d}\boldsymbol{x}=\mathbb{E}\left[{\theta|\boldsymbol{x}}\right] \end{align} In signal process domain, $p(\theta|\boldsymbol{x})$ is called as posterior distribution.


[1] Sengijpta S K. Fundamentals of statistical signal processing: Estimation theory[J]. 1995.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.