Bayesian Estimation Derivation

Question

I am trying to understand Bayesian estimation and I come across this line in my lecture notes:

 θ(Bayesian) = E_θ|x[θ] =  E[π(θ|x)]

So it's meant to reader that the Bayesian estimator is the Conditional Expectation of the sample (x's) which equals the expectation of the posterior (3'rd expression). I understand the derivation up to this point but I dont see how the conditional expectation of theta:

So the middle expression "E_θ|x[θ]" (should look like E underscore θ|x of θ) is:

Integral [θ · π(θ|x) dθ]

and somehow that equals the expectation of the Posterior i.e.

E[π(θ|x)]

Please help! I am hoping it is a simple answer cause there is no explanation between these steps.

Answer 1

At the begin, you should make sure what is the Bayesian estimator. In chapter 11 of [1], Bayesian estimator is defined as an estimator which minimizes the Bayesian mean square error (Bayesian MSE). Let $\boldsymbol{x}$ be an observed signal, $\theta$ is estimated signal and $\hat{\theta}$ is estimator of $\theta$. With those knowledge, we have \begin{align} \text{Bmse}\ (\hat{\theta}) &=\int (\hat{\theta}-\theta)^2p(\boldsymbol{x},\theta)\text{d}\boldsymbol{x}\text{d}\theta\\ &=\int \left[{\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}x}\right]p(\theta)\text{d}\theta \end{align} Thanks to the non-negativity of $p(\theta)$, the minimum of Bayesian MSE can be touched via minimizing inner integral. To this end, we take partial derivative of inner integral w.r.t $\hat{\theta}$ \begin{align} \frac{\partial }{\partial \hat{\theta}}\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}\boldsymbol{x}=-2\int \theta p(\theta|\boldsymbol{x})\text{d}\theta+2\hat{\theta}\int p(\theta|\boldsymbol{x})\text{d}\theta \end{align} Let it be 0 yields \begin{align} \hat{\theta}=\int \theta p(\theta|\boldsymbol{x})\text{d}\boldsymbol{x}=\mathbb{E}\left[{\theta|\boldsymbol{x}}\right] \end{align} In signal process domain, $p(\theta|\boldsymbol{x})$ is called as posterior distribution.

References

[1] Sengijpta S K. Fundamentals of statistical signal processing: Estimation theory[J]. 1995.

Answer 2

You first need to take into account that Bayesian Inference derives itself from decision theory.

Second, in Bayesian inference the point estimator $\hat{\theta}$ is the value that minimizes the Expected Loss Function.

So, for instance, let $\pi(\theta | X_{(n)})$ be the posterior distribution of $\theta$ given the sample $X_{(n)}$ and $\mathcal{L}(\theta, \hat{\theta})$ the Loss function.

So given this two definitions $\hat{\theta}$ will be the value tha minimizes the Expected Loss given by $$\int_{\Theta} \mathcal{L}(\theta, \hat{\theta}) \pi(\theta | X_{(n)})d\theta =\mathbb{E}_{\theta}[L(\theta, \hat{\theta})|X_{(n)}]$$.

Solving this integral for different loss functions yields different point estimators for $\hat{\theta}$, i.e.

• $\mathcal{L}= (\theta - \hat{\theta})^2$ yields the expected value/mean of the posterior distribution $\pi$, i.e. $E_{\theta}[\theta|X_{(n)}]$ which is the BMSE in the answer above.
• $\mathcal{L}= |\theta - \hat{\theta}|$ yields the median of the posterior distribution $\pi$.
• $\mathcal{L}= \left\{ \begin{array}{rcl} 1 & \mbox{for} & |\theta - \hat{\theta}|<\varepsilon \\ 0 & \mbox{for} & |\theta - \hat{\theta}|>\varepsilon \end{array}\right.$ yields the mode of the posterior posterior $\pi$.

However you can construct any loss function that correctly represents preference in your particular inference decision problem.

I really recommend to you looking at this introductory course in bayesian statistics by Manuel Mendoza to get some real graps on bayesian theory and the axioms underlying Bayesian Inference.