# Bayesian Estimation Derivation

I am trying to understand Bayesian estimation and I come across this line in my lecture notes:

 θ(Bayesian) = E_θ|x[θ] =  E[π(θ|x)]


So it's meant to reader that the Bayesian estimator is the Conditional Expectation of the sample (x's) which equals the expectation of the posterior (3'rd expression). I understand the derivation up to this point but I dont see how the conditional expectation of theta:

So the middle expression "E_θ|x[θ]" (should look like E underscore θ|x of θ) is:

Integral [θ · π(θ|x) dθ]

and somehow that equals the expectation of the Posterior i.e.

E[π(θ|x)]


• I've tried thinking about it as E_Y|X (y) = E [ f_Y|X (y|x) ] if that helps but I haven't been able to find a solution, probably because I don't know how to deal with the RHS expectation of a conditional density.... – Mauro Augusto May 20 '16 at 13:36
• I guess its a matter of notation. Let $\pi(\theta |x)$ be the posterior, then the posterior mean (which is the Bayesian estimator), denoted as $\mathbb{E}[\pi(\theta |x)]$, is given by $$\int \theta \ \pi(\theta | x)d\theta.$$ The above is simply the conditional expectation of $\theta$ w.r.t the observations $x$ and this is exactly the posterior mean. – Cavents May 20 '16 at 14:30
• I get that the expression you wrote out is the conditional expectation, thats what a bunch of the derivation leads up to. But how does that equal to the Expectation of the posterior. I mean thats what we are trying to prove isnt it? It seems to me like you are using the result of the proof in the proof. – Mauro Augusto May 20 '16 at 15:25
• Hi -- welcome to math.SE! Here's a reference and tutorial for typesetting math on this site. – joriki May 20 '16 at 16:43
• @MauroAugusto: because $\pi(\theta|x)$ is the posterior density which is a function of $\theta$. Therefore, the integral is exactly the definition of the mean of the posterior (.... recall the definition of the expected value of a random variable). – Cavents May 20 '16 at 17:24

At the begin, you should make sure what is the Bayesian estimator. In chapter 11 of [1], Bayesian estimator is defined as an estimator which minimizes the Bayesian mean square error (Bayesian MSE). Let $$\boldsymbol{x}$$ be an observed signal, $$\theta$$ is estimated signal and $$\hat{\theta}$$ is estimator of $$\theta$$. With those knowledge, we have \begin{align} \text{Bmse}\ (\hat{\theta}) &=\int (\hat{\theta}-\theta)^2p(\boldsymbol{x},\theta)\text{d}\boldsymbol{x}\text{d}\theta\\ &=\int \left[{\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}x}\right]p(\theta)\text{d}\theta \end{align} Thanks to the non-negativity of $$p(\theta)$$, the minimum of Bayesian MSE can be touched via minimizing inner integral. To this end, we take partial derivative of inner integral w.r.t $$\hat{\theta}$$ \begin{align} \frac{\partial }{\partial \hat{\theta}}\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}\boldsymbol{x}=-2\int \theta p(\theta|\boldsymbol{x})\text{d}\theta+2\hat{\theta}\int p(\theta|\boldsymbol{x})\text{d}\theta \end{align} Let it be 0 yields \begin{align} \hat{\theta}=\int \theta p(\theta|\boldsymbol{x})\text{d}\boldsymbol{x}=\mathbb{E}\left[{\theta|\boldsymbol{x}}\right] \end{align} In signal process domain, $$p(\theta|\boldsymbol{x})$$ is called as posterior distribution.