3
$\begingroup$

I am trying to understand Bayesian estimation and I come across this line in my lecture notes:

 θ(Bayesian) = E_θ|x[θ] =  E[π(θ|x)]

So it's meant to reader that the Bayesian estimator is the Conditional Expectation of the sample (x's) which equals the expectation of the posterior (3'rd expression). I understand the derivation up to this point but I dont see how the conditional expectation of theta:

So the middle expression "E_θ|x[θ]" (should look like E underscore θ|x of θ) is:

Integral [θ · π(θ|x) dθ]

and somehow that equals the expectation of the Posterior i.e.

E[π(θ|x)]

Please help! I am hoping it is a simple answer cause there is no explanation between these steps.

$\endgroup$
9
  • $\begingroup$ I've tried thinking about it as E_Y|X (y) = E [ f_Y|X (y|x) ] if that helps but I haven't been able to find a solution, probably because I don't know how to deal with the RHS expectation of a conditional density.... $\endgroup$ May 20 '16 at 13:36
  • $\begingroup$ I guess its a matter of notation. Let $\pi(\theta |x)$ be the posterior, then the posterior mean (which is the Bayesian estimator), denoted as $\mathbb{E}[\pi(\theta |x)]$, is given by $$\int \theta \ \pi(\theta | x)d\theta.$$ The above is simply the conditional expectation of $\theta$ w.r.t the observations $x$ and this is exactly the posterior mean. $\endgroup$
    – Cavents
    May 20 '16 at 14:30
  • $\begingroup$ I get that the expression you wrote out is the conditional expectation, thats what a bunch of the derivation leads up to. But how does that equal to the Expectation of the posterior. I mean thats what we are trying to prove isnt it? It seems to me like you are using the result of the proof in the proof. $\endgroup$ May 20 '16 at 15:25
  • $\begingroup$ Hi -- welcome to math.SE! Here's a reference and tutorial for typesetting math on this site. $\endgroup$
    – joriki
    May 20 '16 at 16:43
  • $\begingroup$ @MauroAugusto: because $\pi(\theta|x)$ is the posterior density which is a function of $\theta$. Therefore, the integral is exactly the definition of the mean of the posterior (.... recall the definition of the expected value of a random variable). $\endgroup$
    – Cavents
    May 20 '16 at 17:24
2
$\begingroup$

At the begin, you should make sure what is the Bayesian estimator. In chapter 11 of [1], Bayesian estimator is defined as an estimator which minimizes the Bayesian mean square error (Bayesian MSE). Let $\boldsymbol{x}$ be an observed signal, $\theta$ is estimated signal and $\hat{\theta}$ is estimator of $\theta$. With those knowledge, we have \begin{align} \text{Bmse}\ (\hat{\theta}) &=\int (\hat{\theta}-\theta)^2p(\boldsymbol{x},\theta)\text{d}\boldsymbol{x}\text{d}\theta\\ &=\int \left[{\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}x}\right]p(\theta)\text{d}\theta \end{align} Thanks to the non-negativity of $p(\theta)$, the minimum of Bayesian MSE can be touched via minimizing inner integral. To this end, we take partial derivative of inner integral w.r.t $\hat{\theta}$ \begin{align} \frac{\partial }{\partial \hat{\theta}}\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}\boldsymbol{x}=-2\int \theta p(\theta|\boldsymbol{x})\text{d}\theta+2\hat{\theta}\int p(\theta|\boldsymbol{x})\text{d}\theta \end{align} Let it be 0 yields \begin{align} \hat{\theta}=\int \theta p(\theta|\boldsymbol{x})\text{d}\boldsymbol{x}=\mathbb{E}\left[{\theta|\boldsymbol{x}}\right] \end{align} In signal process domain, $p(\theta|\boldsymbol{x})$ is called as posterior distribution.

References

[1] Sengijpta S K. Fundamentals of statistical signal processing: Estimation theory[J]. 1995.

$\endgroup$
1
$\begingroup$

You first need to take into account that Bayesian Inference derives itself from decision theory.

Second, in Bayesian inference the point estimator $\hat{\theta}$ is the value that minimizes the Expected Loss Function.

So, for instance, let $\pi(\theta | X_{(n)})$ be the posterior distribution of $\theta$ given the sample $X_{(n)}$ and $\mathcal{L}(\theta, \hat{\theta})$ the Loss function.

So given this two definitions $\hat{\theta}$ will be the value tha minimizes the Expected Loss given by $$\int_{\Theta} \mathcal{L}(\theta, \hat{\theta}) \pi(\theta | X_{(n)})d\theta =\mathbb{E}_{\theta}[L(\theta, \hat{\theta})|X_{(n)}]$$.

Solving this integral for different loss functions yields different point estimators for $\hat{\theta}$, i.e.

  • $\mathcal{L}= (\theta - \hat{\theta})^2$ yields the expected value/mean of the posterior distribution $\pi$, i.e. $E_{\theta}[\theta|X_{(n)}]$ which is the BMSE in the answer above.
  • $\mathcal{L}= |\theta - \hat{\theta}|$ yields the median of the posterior distribution $\pi$.
  • $\mathcal{L}= \left\{ \begin{array}{rcl} 1 & \mbox{for} & |\theta - \hat{\theta}|<\varepsilon \\ 0 & \mbox{for} & |\theta - \hat{\theta}|>\varepsilon \end{array}\right.$ yields the mode of the posterior posterior $\pi$.

However you can construct any loss function that correctly represents preference in your particular inference decision problem.

I really recommend to you looking at this introductory course in bayesian statistics by Manuel Mendoza to get some real graps on bayesian theory and the axioms underlying Bayesian Inference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.