3
$\begingroup$

I am trying to understand Bayesian estimation and I come across this line in my lecture notes:

 θ(Bayesian) = E_θ|x[θ] =  E[π(θ|x)]

So it's meant to reader that the Bayesian estimator is the Conditional Expectation of the sample (x's) which equals the expectation of the posterior (3'rd expression). I understand the derivation up to this point but I dont see how the conditional expectation of theta:

So the middle expression "E_θ|x[θ]" (should look like E underscore θ|x of θ) is:

Integral [θ · π(θ|x) dθ]

and somehow that equals the expectation of the Posterior i.e.

E[π(θ|x)]

Please help! I am hoping it is a simple answer cause there is no explanation between these steps.

$\endgroup$
9
  • $\begingroup$ I've tried thinking about it as E_Y|X (y) = E [ f_Y|X (y|x) ] if that helps but I haven't been able to find a solution, probably because I don't know how to deal with the RHS expectation of a conditional density.... $\endgroup$ Commented May 20, 2016 at 13:36
  • $\begingroup$ I guess its a matter of notation. Let $\pi(\theta |x)$ be the posterior, then the posterior mean (which is the Bayesian estimator), denoted as $\mathbb{E}[\pi(\theta |x)]$, is given by $$\int \theta \ \pi(\theta | x)d\theta.$$ The above is simply the conditional expectation of $\theta$ w.r.t the observations $x$ and this is exactly the posterior mean. $\endgroup$
    – Cavents
    Commented May 20, 2016 at 14:30
  • $\begingroup$ I get that the expression you wrote out is the conditional expectation, thats what a bunch of the derivation leads up to. But how does that equal to the Expectation of the posterior. I mean thats what we are trying to prove isnt it? It seems to me like you are using the result of the proof in the proof. $\endgroup$ Commented May 20, 2016 at 15:25
  • $\begingroup$ Hi -- welcome to math.SE! Here's a reference and tutorial for typesetting math on this site. $\endgroup$
    – joriki
    Commented May 20, 2016 at 16:43
  • $\begingroup$ @MauroAugusto: because $\pi(\theta|x)$ is the posterior density which is a function of $\theta$. Therefore, the integral is exactly the definition of the mean of the posterior (.... recall the definition of the expected value of a random variable). $\endgroup$
    – Cavents
    Commented May 20, 2016 at 17:24

2 Answers 2

3
$\begingroup$

At the begin, you should make sure what is the Bayesian estimator. In chapter 11 of [1], Bayesian estimator is defined as an estimator which minimizes the Bayesian mean square error (Bayesian MSE). Let $\boldsymbol{x}$ be an observed signal, $\theta$ is estimated signal and $\hat{\theta}$ is estimator of $\theta$. With those knowledge, we have \begin{align} \text{Bmse}\ (\hat{\theta}) &=\int (\hat{\theta}-\theta)^2p(\boldsymbol{x},\theta)\text{d}\boldsymbol{x}\text{d}\theta\\ &=\int \left[{\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}x}\right]p(\theta)\text{d}\theta \end{align} Thanks to the non-negativity of $p(\theta)$, the minimum of Bayesian MSE can be touched via minimizing inner integral. To this end, we take partial derivative of inner integral w.r.t $\hat{\theta}$ \begin{align} \frac{\partial }{\partial \hat{\theta}}\int (\hat{\theta}-\theta)^2p(\boldsymbol{x}|\theta)\text{d}\boldsymbol{x}=-2\int \theta p(\theta|\boldsymbol{x})\text{d}\theta+2\hat{\theta}\int p(\theta|\boldsymbol{x})\text{d}\theta \end{align} Let it be 0 yields \begin{align} \hat{\theta}=\int \theta p(\theta|\boldsymbol{x})\text{d}\boldsymbol{x}=\mathbb{E}\left[{\theta|\boldsymbol{x}}\right] \end{align} In signal process domain, $p(\theta|\boldsymbol{x})$ is called as posterior distribution.

References

[1] Sengijpta S K. Fundamentals of statistical signal processing: Estimation theory[J]. 1995.

$\endgroup$
1
$\begingroup$

You first need to take into account that Bayesian Inference derives itself from decision theory.

Second, in Bayesian inference the point estimator $\hat{\theta}$ is the value that minimizes the Expected Loss Function.

So, for instance, let $\pi(\theta | X_{(n)})$ be the posterior distribution of $\theta$ given the sample $X_{(n)}$ and $\mathcal{L}(\theta, \hat{\theta})$ the Loss function.

So given this two definitions $\hat{\theta}$ will be the value tha minimizes the Expected Loss given by $$\int_{\Theta} \mathcal{L}(\theta, \hat{\theta}) \pi(\theta | X_{(n)})d\theta =\mathbb{E}_{\theta}[L(\theta, \hat{\theta})|X_{(n)}]$$.

Solving this integral for different loss functions yields different point estimators for $\hat{\theta}$, i.e.

  • $\mathcal{L}= (\theta - \hat{\theta})^2$ yields the expected value/mean of the posterior distribution $\pi$, i.e. $E_{\theta}[\theta|X_{(n)}]$ which is the BMSE in the answer above.
  • $\mathcal{L}= |\theta - \hat{\theta}|$ yields the median of the posterior distribution $\pi$.
  • $\mathcal{L}= \left\{ \begin{array}{rcl} 1 & \mbox{for} & |\theta - \hat{\theta}|<\varepsilon \\ 0 & \mbox{for} & |\theta - \hat{\theta}|>\varepsilon \end{array}\right.$ yields the mode of the posterior posterior $\pi$.

However you can construct any loss function that correctly represents preference in your particular inference decision problem.

I really recommend to you looking at this introductory course in bayesian statistics by Manuel Mendoza to get some real graps on bayesian theory and the axioms underlying Bayesian Inference.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .