I have a Gaussian mixture model, given by: $$ x \sim \sum_{i = 1}^M \alpha_i N(\mu_i, C_i) $$

Is there a way I can compute the overall covariance matrix if $x$? I would like to say "$x$ has a covariance matrix given by $C$".

  • 2
    $\begingroup$ Is $x$ a vector, or a scalar random variable? $\endgroup$ – Sasha Sep 14 '12 at 21:50
  • $\begingroup$ Please avoid using acronyms. Using expanded names allows for ease of searching, and invites those uninitiated to look the term up and learn about it. $\endgroup$ – Sasha Sep 14 '12 at 21:59
  • $\begingroup$ Removed the tag gauss-sum as that refers to a different concept. $\endgroup$ – Sasha Sep 14 '12 at 22:00
  • $\begingroup$ My answer is at least simpler than the others. But also, I think it's the more-or-less natural way to do it and uses a technique that ought to become habitual for anyone who works with expectations and variances. $\endgroup$ – Michael Hardy Sep 15 '12 at 2:13
  • $\begingroup$ @shasha : vectors.. sorry about that $\endgroup$ – NSR Sep 16 '12 at 1:42

$\newcommand{\var}{\operatorname{var}}$ You can write $x = y + \text{error}$, where $y = \mu_i$ with probability $\alpha_i$, for $i=1,\ldots,M$, and the conditional probability distribution of the "error" given $y$ is $N(0,C_i)$. Then we have $$ E(x) = E(E(x\mid y)) = E\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} = \sum_{i=1}^M\alpha_i\mu_i, $$ and $$ \begin{align} \var(x) = {} & E(\var(x\mid y)) + \var(E(x \mid y)) \\[12pt] = {} & E\left.\begin{cases} \vdots \\ C_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} \\ & {} + \var\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases} \right\} \\[12pt] = {} & \sum_{i=1}^M \alpha_i C_i + \sum_{i=1}^M \alpha_i(\mu_i-\bar\mu)(\mu_i-\bar\mu)^T, \end{align} $$ where $\bar\mu=\sum_{i=1}^M \alpha_i\mu_i$.

| cite | improve this answer | |
  • $\begingroup$ Aren't you missing probabilities $\alpha_i$ in the last two summations in your answer? You seem to be using $\alpha_i = M^{-1}$ for all $i$. $\endgroup$ – Dilip Sarwate Sep 15 '12 at 13:19
  • $\begingroup$ @DilipSarwate : Fixed. Yes, it's weighted average with generally unequal weights. $\endgroup$ – Michael Hardy Sep 15 '12 at 18:56
  • $\begingroup$ @MichaelHardy So, here $var(x)$ is the covariance matrix user is looking at, am I right? $\endgroup$ – Autonomous May 16 '15 at 5:26
  • $\begingroup$ Yes. ${}\qquad{}$ $\endgroup$ – Michael Hardy May 16 '15 at 6:45
  • $\begingroup$ @MichaelHardy If you may clear one more doubt of mine regarding covariance matrices, that would be great. Thanks for replying promptly yesterday. $\endgroup$ – Autonomous May 16 '15 at 19:57

A mixture model commonly refers to a weighted sum of densities, not a weighted sum of random variables as in Sasha's answer As a simplest example, a (scalar) random variable $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z)=\alpha f_X(z)+(1−\alpha)f_Y(z), ~0<\alpha<1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2)\neq(\mu_Y,\sigma_Y^2)$. It follows straightforwardly that $$\begin{aligned} E[Z]&= \int_{-\infty}^\infty zf_Z(z)\,\mathrm dz& &= \alpha\mu_X + (1-\alpha)\mu_Y\\ E[Z^2]&= \int_{-\infty}^\infty z^2f_Z(z)\,\mathrm dz& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2)\\ \text{var}(Z) &= E[Z^2] - (E[Z])^2& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2 \end{aligned}$$ which unfortunately does not simplify to a pretty formula.

More generally, a mixture density would have $n, n > 1,$ terms with positive weights $\alpha_1, \alpha_2, \ldots, \alpha_n$ summing to $1$. It is simplest to think of a partition of the sample space into events $A_k, 1 \leq k \leq n$, with $P(A_k) = \alpha_k$. Then, $Z$ is a random variable whose conditional distribution given $A_k$ is a Gaussian distribution $f_k(z) \sim N(\mu_k,\sigma_k^2)$, and thus the unconditional distribution is, via the law of total probability, $$f(z) = \sum_{k=1}^n \alpha_k f_k(z).$$ The ungainly expression $\alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2$ derived earlier for the variance of $Z$ can thus be manipulated into $$\bigr[\alpha\sigma_X^2 + (1-\alpha)\sigma_Y^2\bigr] + \biggr(\bigr[\alpha\mu_X^2 + (1-\alpha)\mu_Y^2\bigr] - \bigr[\alpha\mu_X + (1-\alpha)\mu_Y\bigr]^2\biggr)$$ which, while still not pretty, can be identified as an illustration of the conditional variance formula:

The variance of $Z$ is the mean of the conditional variance plus the variance of the conditional mean.

When $Z$ is a vector random variable whose conditional distributions are jointly Gaussian with mean vector $\mu$ and covariance matrix $C_i$, similar calculations can be done, and the unconditional covariance matrix computed. Suppose that the conditional density of $Z$ given $A_k$ is a jointly Gaussian density with mean vector $\mu^{(k)}$ and covariance matrix $C^{(k)}$. Then, $$E[Z_i] = \sum_{k=1}^n \alpha_k \mu_i^{(k)} ~\text{and}~ E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$$ giving $$\begin{align}C_{i,j} = \text{cov}(Z_i,Z_j) &= \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr) - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\\ &= \left(\sum_{k=1}^n \alpha_k C_{i,j}^{(k)}\right) + \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\mu_j^{(k)} - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\right) \end{align}$$ which is an illustration of the conditional covariance formula:

The covariance of two random variables is the mean of the conditional covariances plus the covariance of the conditional means.

| cite | improve this answer | |
  • 1
    $\begingroup$ Could you explain how you get $E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$ ? Thank you $\endgroup$ – Pop Jul 12 '13 at 9:21
  • 1
    $\begingroup$ @Pop $E[Z_i Z_j] = E_A[ E[ Z_i Z_j | A ] ] = E_A[ Cov[Z_i|A, Z_j|A] + E[Z_i|A] E[Z_j|A] ]$ $\endgroup$ – tflutre Jul 28 '14 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.