24
$\begingroup$

I have a Gaussian mixture model, given by: $$ X \sim \sum_{i = 1}^M \alpha_i N_p(\mu_i, C_i) $$ such that $\sum_{i=1}^M\alpha_i =1 $. Is there a way I can compute the overall covariance matrix if $x$? I would like to say "$X$ has a covariance matrix given by $C$".

$\endgroup$
5
  • 2
    $\begingroup$ Is $x$ a vector, or a scalar random variable? $\endgroup$
    – Sasha
    Commented Sep 14, 2012 at 21:50
  • $\begingroup$ Please avoid using acronyms. Using expanded names allows for ease of searching, and invites those uninitiated to look the term up and learn about it. $\endgroup$
    – Sasha
    Commented Sep 14, 2012 at 21:59
  • 1
    $\begingroup$ Removed the tag gauss-sum as that refers to a different concept. $\endgroup$
    – Sasha
    Commented Sep 14, 2012 at 22:00
  • $\begingroup$ My answer is at least simpler than the others. But also, I think it's the more-or-less natural way to do it and uses a technique that ought to become habitual for anyone who works with expectations and variances. $\endgroup$ Commented Sep 15, 2012 at 2:13
  • $\begingroup$ @shasha : vectors.. sorry about that $\endgroup$
    – NSR
    Commented Sep 16, 2012 at 1:42

3 Answers 3

20
$\begingroup$

$\newcommand{\var}{\operatorname{var}}$ You can write $x = y + \text{error}$, where $y = \mu_i$ with probability $\alpha_i$, for $i=1,\ldots,M$, and the conditional probability distribution of the "error" given $y$ is $N(0,C_i)$. Then we have $$ E(x) = E(E(x\mid y)) = E\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} = \sum_{i=1}^M\alpha_i\mu_i, $$ and $$ \begin{align} \var(x) = {} & E(\var(x\mid y)) + \var(E(x \mid y)) \\[12pt] = {} & E\left.\begin{cases} \vdots \\ C_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} \\ & {} + \var\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases} \right\} \\[12pt] = {} & \sum_{i=1}^M \alpha_i C_i + \sum_{i=1}^M \alpha_i(\mu_i-\bar\mu)(\mu_i-\bar\mu)^T, \end{align} $$ where $\bar\mu=\sum_{i=1}^M \alpha_i\mu_i$.

$\endgroup$
5
  • $\begingroup$ Aren't you missing probabilities $\alpha_i$ in the last two summations in your answer? You seem to be using $\alpha_i = M^{-1}$ for all $i$. $\endgroup$ Commented Sep 15, 2012 at 13:19
  • $\begingroup$ @DilipSarwate : Fixed. Yes, it's weighted average with generally unequal weights. $\endgroup$ Commented Sep 15, 2012 at 18:56
  • $\begingroup$ @MichaelHardy So, here $var(x)$ is the covariance matrix user is looking at, am I right? $\endgroup$
    – Autonomous
    Commented May 16, 2015 at 5:26
  • $\begingroup$ Yes. ${}\qquad{}$ $\endgroup$ Commented May 16, 2015 at 6:45
  • $\begingroup$ @MichaelHardy If you may clear one more doubt of mine regarding covariance matrices, that would be great. Thanks for replying promptly yesterday. $\endgroup$
    – Autonomous
    Commented May 16, 2015 at 19:57
18
$\begingroup$

A mixture model commonly refers to a weighted sum of densities, not a weighted sum of random variables as in Sasha's answer As a simplest example, a (scalar) random variable $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z)=\alpha f_X(z)+(1−\alpha)f_Y(z), ~0<\alpha<1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2)\neq(\mu_Y,\sigma_Y^2)$. It follows straightforwardly that $$\begin{aligned} E[Z]&= \int_{-\infty}^\infty zf_Z(z)\,\mathrm dz& &= \alpha\mu_X + (1-\alpha)\mu_Y\\ E[Z^2]&= \int_{-\infty}^\infty z^2f_Z(z)\,\mathrm dz& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2)\\ \text{var}(Z) &= E[Z^2] - (E[Z])^2& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2 \end{aligned}$$ which unfortunately does not simplify to a pretty formula.

More generally, a mixture density would have $n, n > 1,$ terms with positive weights $\alpha_1, \alpha_2, \ldots, \alpha_n$ summing to $1$. It is simplest to think of a partition of the sample space into events $A_k, 1 \leq k \leq n$, with $P(A_k) = \alpha_k$. Then, $Z$ is a random variable whose conditional distribution given $A_k$ is a Gaussian distribution $f_k(z) \sim N(\mu_k,\sigma_k^2)$, and thus the unconditional distribution is, via the law of total probability, $$f(z) = \sum_{k=1}^n \alpha_k f_k(z).$$ The ungainly expression $\alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2$ derived earlier for the variance of $Z$ can thus be manipulated into $$\bigr[\alpha\sigma_X^2 + (1-\alpha)\sigma_Y^2\bigr] + \biggr(\bigr[\alpha\mu_X^2 + (1-\alpha)\mu_Y^2\bigr] - \bigr[\alpha\mu_X + (1-\alpha)\mu_Y\bigr]^2\biggr)$$ which, while still not pretty, can be identified as an illustration of the conditional variance formula:

The variance of $Z$ is the mean of the conditional variance plus the variance of the conditional mean.

When $Z$ is a vector random variable whose conditional distributions are jointly Gaussian with mean vector $\mu$ and covariance matrix $C_i$, similar calculations can be done, and the unconditional covariance matrix computed. Suppose that the conditional density of $Z$ given $A_k$ is a jointly Gaussian density with mean vector $\mu^{(k)}$ and covariance matrix $C^{(k)}$. Then, $$E[Z_i] = \sum_{k=1}^n \alpha_k \mu_i^{(k)} ~\text{and}~ E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$$ giving $$\begin{align}C_{i,j} = \text{cov}(Z_i,Z_j) &= \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr) - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\\ &= \left(\sum_{k=1}^n \alpha_k C_{i,j}^{(k)}\right) + \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\mu_j^{(k)} - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\right) \end{align}$$ which is an illustration of the conditional covariance formula:

The covariance of two random variables is the mean of the conditional covariances plus the covariance of the conditional means.

$\endgroup$
2
  • 1
    $\begingroup$ Could you explain how you get $E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$ ? Thank you $\endgroup$
    – Pop
    Commented Jul 12, 2013 at 9:21
  • 1
    $\begingroup$ @Pop $E[Z_i Z_j] = E_A[ E[ Z_i Z_j | A ] ] = E_A[ Cov[Z_i|A, Z_j|A] + E[Z_i|A] E[Z_j|A] ]$ $\endgroup$
    – tflutre
    Commented Jul 28, 2014 at 12:36
6
$\begingroup$

Motivated by the answer provided by Michael Hardy, a formal solution to such question might be formulated as follows:

By introducing a new hidden variable $I$ to represent the identity of the local model, the probability of the Gaussian mixtures can be decomposed as: $$p(x|I=i)=\mathcal{N}(\mu_i,C_i)$$ $$p(I=i)=\alpha_i$$ Therefore, $$E(x)=E[E(x|I=i)]=\sum_{i=1}^{M} \alpha_i \mu_i$$ \begin{align} Var(x)&=E[Var(x|I=i)]+Var[E(x|I=i)] \\ &=\sum_{i=1}^{M} \alpha_i C_i + \sum_{i=1}^{M} \alpha_i (\mu_i-\bar{\mu})(\mu_i-\bar{\mu})^T \end{align} where $\bar{\mu}=E(x)$. Moreover, the Law of total expectation and Law of total variance have been used in above two equations.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .