5
$\begingroup$

Assume we have a Gaussian distribution $p(x) \sim \mathcal{N}(\mu_p,\Sigma_p)$

For any point $X$, it is easy to compute the density of $x$ in $p$: $$p(x) = \frac{1}{|2\pi \Sigma_p|^\frac{1}{2}}e^{-\frac{1}{2}(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)}$$

enter image description here

Now suppose that we have another Gaussian distribution $q(x) \sim \mathcal{N}(\mu_q,\Sigma_q)$

** What is the expectation of q(x) in p(x) . I mean, if we randomly sample a point $x$ from P, what is the **expected probability of x in Q?

enter image description here

NOTE: I need the solution in closed form.

Thanks!

$\endgroup$
  • 2
    $\begingroup$ The right side of $$p(x) = \frac{1}{|2\pi \Sigma_i|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma_i^{-1}(x-\mu)}dx$$ is not "the probability of $x$ in $p$" but the value of the density of a bivariate normal random variable at $x$ (and the density value might well exceed $1$). In "If we randomly sample a point $x$ from (a different bivariate normal distribution) $Q$" there is no such thing as the "expected probability of $x$ in $P$": the value of the density (assuming the distribution is $P$) at $x$ does not depend on how $x$ was chosen, e.g. with distribution $Q$. $\endgroup$ – Dilip Sarwate Jun 2 '14 at 2:38
  • 2
    $\begingroup$ I think what the OP is asking is this: if you draw sample $x$ from pdf $p(x)$, then what is the expected value of the function $q(x)$, which is by the way a multivariate normal i.e.: $$q(x) = \frac{1}{|2\pi \Sigma_i|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)}$$, $\Sigma_q$ is positive definite, and we have $$\int_{x\in \mathcal{X}}q(x)dx=1$$ I agree that the question could be phrased more clearly, but I think both the question, and the given answer make sense. Unless, the OP meant something else! $\endgroup$ – Alt Jun 2 '14 at 5:25
  • $\begingroup$ Well, the only thing that is clear to me is that the OP has not phrase the question in a mathematically precise way. Accepting such an interpretation, the OP actually asked the following: Given two independent multivariate normal random variables $\boldsymbol P$ and $\boldsymbol Q$, what is the expected value of the density of $\boldsymbol P$ with respect to $\boldsymbol Q$, not the other way around. That is, we want to find ${\rm E}[f_{\boldsymbol P}(\boldsymbol Q)]$. $\endgroup$ – heropup Jun 2 '14 at 5:34
  • 2
    $\begingroup$ @heropup: Exactly, Although in this case we have: $${\rm E}_{x\sim p}[q(x)]={\rm E}_{x\sim q}[p(x)]=\int_{x\in \mathcal{X}}q(x)p(x)dx$$ So, like I said, unless the OP means something else the answer below is correct. $\endgroup$ – Alt Jun 2 '14 at 5:38
  • $\begingroup$ Thank you all guys for both your comments and edits! Yes, I meant exactly what you said. (Sorry, I'm a CS major, so I'm sometimes confused with the true mathematical terminology). So what is the exact mathematical notation for what I meant? ${\rm E}[f_{\boldsymbol P}(\boldsymbol Q)]$ or ${\rm E}_{x\sim p}[q(x)]$ or ${\rm E}_{x\sim q}[p(x)]$? $\endgroup$ – AliHadian Jun 2 '14 at 6:11
1
$\begingroup$

Note that in this case we will have:

$$E_{x\sim p(x)}[q(x)]=E_{x\sim q(x)}[p(x)]=\int_{x\in \mathcal{X}} p(x)q(x)dx$$ $$=\int_{x\in \mathcal{X}} \left(\frac{1}{|2\pi \Sigma_p|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)}\right)\left(\frac{1}{|2\pi \Sigma_q|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)}\right)dx$$

$$=\int_{x\in \mathcal{X}} \frac{1}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}}e^{-\frac{1}{2}\left((x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)\right)}dx$$

Now go ahead and, expand $(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)$ and regroup the the terms to get: $$(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)=x^TAx+b^Tx+d$$

Then work out the algebra to get $\Sigma$ and $\mu$ such that:

$$x^TAx+b^Tx+d=(x-\mu)^T\Sigma^{-1}(x-\mu)+F$$

After this you will have some thing like this:

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} \int_{x\in \mathcal{X}}\frac{1}{|2\pi \Sigma|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}dx$$

$\int_{x\in \mathcal{X}}\frac{1}{|2\pi \Sigma|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}dx=1$, so you'll have:

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} $$

Edit

$$(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)=x^T(\Sigma_p^{-1}+\Sigma_q^{-1})x+(-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})x+(\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q)$$

Let $A=\Sigma_p^{-1}+\Sigma_q^{-1}$,$b=(-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})^T$, and $d=\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q$.

Now we should find $\Sigma$, $\mu$, and $F$, we have :

$$(x-\mu)^T\Sigma^{-1}(x-\mu)=x^T\Sigma x-2\mu^T\Sigma^{-1}x+\mu^T \Sigma^{-1} \mu $$

Let: $$x^TAx+b^Tx+d=x^T\Sigma^{-1} x-2\mu^T\Sigma^{-1}x+\mu^T \Sigma^{-1} \mu +F$$

$$\Sigma^{-1}=A=\Sigma_p^{-1}+\Sigma_q^{-1}\Rightarrow \Sigma=\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1}$$

$$b^{T}=-2\mu^T\Sigma^{-1}\Rightarrow \mu=-\frac{1}{2}\Sigma b=-\frac{1}{2}\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1} (-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})^T$$

$$\Rightarrow \mu=\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1} (\Sigma_p^{-1}\mu_p+\Sigma_q^{-1}\mu_q)$$

$$F=d-\mu^T \Sigma^{-1} \mu$$ $$\Rightarrow F=\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q-\mu^T \Sigma^{-1} \mu$$

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} $$

Note that, the result is just an scalar, please let me know if you have further questions :)

$\endgroup$
  • $\begingroup$ ... So you mean there is no way to compute the expected probability in a simple formula? (It seems that we need an equation-solver program for this?) $\endgroup$ – AliHadian May 29 '14 at 22:59
  • $\begingroup$ $${ (x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q) } $$ $$= x^T (\Sigma_p^{-1} + \Sigma_q^{-1})x -2 ( \mu_p^T \Sigma_p^{-1}+\mu_q^T \Sigma_q^{-1}) x + \left( \mu_p^T \Sigma_p^{-1}\mu_p + \mu_q^T \Sigma_q^{-1}\mu_q \right) $$ so we have: $$A=\Sigma_p^{-1} + \Sigma_q^{-1}$$ $$b^T = -2 ( \mu_p^T \Sigma_p^{-1}+\mu_q^T \Sigma_q^{-1})$$ $$d= \mu_p^T \Sigma_p^{-1}\mu_p + \mu_q^T \Sigma_q^{-1}\mu_q$$ How should I extract $\Sigma$ and $\mu$ from A,b,c??? $\endgroup$ – AliHadian Jun 1 '14 at 6:28
  • 1
    $\begingroup$ Neither the question nor this answer make much sense. I am downvoting both. $\endgroup$ – Dilip Sarwate Jun 1 '14 at 23:06
  • 2
    $\begingroup$ @DilipSarwate: Could you please explain what doesn't make sense? $\endgroup$ – Alt Jun 2 '14 at 2:10
  • 1
    $\begingroup$ @DilipSarwate: Calculation of expected probability is like calculation of expectation of any other function! why it doesn't make sense? And the answer is clear, and carefully derived, what's wrong? $\endgroup$ – Alt Jun 2 '14 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.