Probability of a gaussian distribution in another gaussian distribution

Assume we have a Gaussian distribution $p(x) \sim \mathcal{N}(\mu_p,\Sigma_p)$

For any point $X$, it is easy to compute the density of $x$ in $p$: $$p(x) = \frac{1}{|2\pi \Sigma_p|^\frac{1}{2}}e^{-\frac{1}{2}(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)}$$

Now suppose that we have another Gaussian distribution $q(x) \sim \mathcal{N}(\mu_q,\Sigma_q)$

** What is the expectation of q(x) in p(x) . I mean, if we randomly sample a point $x$ from P, what is the **expected probability of x in Q?

NOTE: I need the solution in closed form.

Thanks!

• The right side of $$p(x) = \frac{1}{|2\pi \Sigma_i|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma_i^{-1}(x-\mu)}dx$$ is not "the probability of $x$ in $p$" but the value of the density of a bivariate normal random variable at $x$ (and the density value might well exceed $1$). In "If we randomly sample a point $x$ from (a different bivariate normal distribution) $Q$" there is no such thing as the "expected probability of $x$ in $P$": the value of the density (assuming the distribution is $P$) at $x$ does not depend on how $x$ was chosen, e.g. with distribution $Q$. – Dilip Sarwate Jun 2 '14 at 2:38
• I think what the OP is asking is this: if you draw sample $x$ from pdf $p(x)$, then what is the expected value of the function $q(x)$, which is by the way a multivariate normal i.e.: $$q(x) = \frac{1}{|2\pi \Sigma_i|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)}$$, $\Sigma_q$ is positive definite, and we have $$\int_{x\in \mathcal{X}}q(x)dx=1$$ I agree that the question could be phrased more clearly, but I think both the question, and the given answer make sense. Unless, the OP meant something else! – Alt Jun 2 '14 at 5:25
• Well, the only thing that is clear to me is that the OP has not phrase the question in a mathematically precise way. Accepting such an interpretation, the OP actually asked the following: Given two independent multivariate normal random variables $\boldsymbol P$ and $\boldsymbol Q$, what is the expected value of the density of $\boldsymbol P$ with respect to $\boldsymbol Q$, not the other way around. That is, we want to find ${\rm E}[f_{\boldsymbol P}(\boldsymbol Q)]$. – heropup Jun 2 '14 at 5:34
• @heropup: Exactly, Although in this case we have: $${\rm E}_{x\sim p}[q(x)]={\rm E}_{x\sim q}[p(x)]=\int_{x\in \mathcal{X}}q(x)p(x)dx$$ So, like I said, unless the OP means something else the answer below is correct. – Alt Jun 2 '14 at 5:38
• Thank you all guys for both your comments and edits! Yes, I meant exactly what you said. (Sorry, I'm a CS major, so I'm sometimes confused with the true mathematical terminology). So what is the exact mathematical notation for what I meant? ${\rm E}[f_{\boldsymbol P}(\boldsymbol Q)]$ or ${\rm E}_{x\sim p}[q(x)]$ or ${\rm E}_{x\sim q}[p(x)]$? – AliHadian Jun 2 '14 at 6:11

Note that in this case we will have:

$$E_{x\sim p(x)}[q(x)]=E_{x\sim q(x)}[p(x)]=\int_{x\in \mathcal{X}} p(x)q(x)dx$$ $$=\int_{x\in \mathcal{X}} \left(\frac{1}{|2\pi \Sigma_p|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)}\right)\left(\frac{1}{|2\pi \Sigma_q|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)}\right)dx$$

$$=\int_{x\in \mathcal{X}} \frac{1}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}}e^{-\frac{1}{2}\left((x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)\right)}dx$$

Now go ahead and, expand $(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)$ and regroup the the terms to get: $$(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)=x^TAx+b^Tx+d$$

Then work out the algebra to get $\Sigma$ and $\mu$ such that:

$$x^TAx+b^Tx+d=(x-\mu)^T\Sigma^{-1}(x-\mu)+F$$

After this you will have some thing like this:

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} \int_{x\in \mathcal{X}}\frac{1}{|2\pi \Sigma|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}dx$$

$\int_{x\in \mathcal{X}}\frac{1}{|2\pi \Sigma|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}dx=1$, so you'll have:

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}}$$

Edit

$$(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)=x^T(\Sigma_p^{-1}+\Sigma_q^{-1})x+(-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})x+(\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q)$$

Let $A=\Sigma_p^{-1}+\Sigma_q^{-1}$,$b=(-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})^T$, and $d=\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q$.

Now we should find $\Sigma$, $\mu$, and $F$, we have :

$$(x-\mu)^T\Sigma^{-1}(x-\mu)=x^T\Sigma x-2\mu^T\Sigma^{-1}x+\mu^T \Sigma^{-1} \mu$$

Let: $$x^TAx+b^Tx+d=x^T\Sigma^{-1} x-2\mu^T\Sigma^{-1}x+\mu^T \Sigma^{-1} \mu +F$$

$$\Sigma^{-1}=A=\Sigma_p^{-1}+\Sigma_q^{-1}\Rightarrow \Sigma=\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1}$$

$$b^{T}=-2\mu^T\Sigma^{-1}\Rightarrow \mu=-\frac{1}{2}\Sigma b=-\frac{1}{2}\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1} (-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})^T$$

$$\Rightarrow \mu=\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1} (\Sigma_p^{-1}\mu_p+\Sigma_q^{-1}\mu_q)$$

$$F=d-\mu^T \Sigma^{-1} \mu$$ $$\Rightarrow F=\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q-\mu^T \Sigma^{-1} \mu$$

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}}$$

Note that, the result is just an scalar, please let me know if you have further questions :)

• ... So you mean there is no way to compute the expected probability in a simple formula? (It seems that we need an equation-solver program for this?) – AliHadian May 29 '14 at 22:59
• $${ (x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q) }$$ $$= x^T (\Sigma_p^{-1} + \Sigma_q^{-1})x -2 ( \mu_p^T \Sigma_p^{-1}+\mu_q^T \Sigma_q^{-1}) x + \left( \mu_p^T \Sigma_p^{-1}\mu_p + \mu_q^T \Sigma_q^{-1}\mu_q \right)$$ so we have: $$A=\Sigma_p^{-1} + \Sigma_q^{-1}$$ $$b^T = -2 ( \mu_p^T \Sigma_p^{-1}+\mu_q^T \Sigma_q^{-1})$$ $$d= \mu_p^T \Sigma_p^{-1}\mu_p + \mu_q^T \Sigma_q^{-1}\mu_q$$ How should I extract $\Sigma$ and $\mu$ from A,b,c??? – AliHadian Jun 1 '14 at 6:28
• Neither the question nor this answer make much sense. I am downvoting both. – Dilip Sarwate Jun 1 '14 at 23:06
• @DilipSarwate: Could you please explain what doesn't make sense? – Alt Jun 2 '14 at 2:10
• @DilipSarwate: Calculation of expected probability is like calculation of expectation of any other function! why it doesn't make sense? And the answer is clear, and carefully derived, what's wrong? – Alt Jun 2 '14 at 5:17