Calculation of the Covariance of Gaussian Mixtures

I have a Gaussian mixture model, given by: $$X \sim \sum_{i = 1}^M \alpha_i N_p(\mu_i, C_i)$$ such that $$\sum_{i=1}^M\alpha_i =1$$. Is there a way I can compute the overall covariance matrix if $$x$$? I would like to say "$$X$$ has a covariance matrix given by $$C$$".

• Is $x$ a vector, or a scalar random variable? Commented Sep 14, 2012 at 21:50
• Please avoid using acronyms. Using expanded names allows for ease of searching, and invites those uninitiated to look the term up and learn about it. Commented Sep 14, 2012 at 21:59
• Removed the tag gauss-sum as that refers to a different concept. Commented Sep 14, 2012 at 22:00
• My answer is at least simpler than the others. But also, I think it's the more-or-less natural way to do it and uses a technique that ought to become habitual for anyone who works with expectations and variances. Commented Sep 15, 2012 at 2:13
• @shasha : vectors.. sorry about that
– NSR
Commented Sep 16, 2012 at 1:42

$$\newcommand{\var}{\operatorname{var}}$$ You can write $$x = y + \text{error}$$, where $$y = \mu_i$$ with probability $$\alpha_i$$, for $$i=1,\ldots,M$$, and the conditional probability distribution of the "error" given $$y$$ is $$N(0,C_i)$$. Then we have $$E(x) = E(E(x\mid y)) = E\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} = \sum_{i=1}^M\alpha_i\mu_i,$$ and \begin{align} \var(x) = {} & E(\var(x\mid y)) + \var(E(x \mid y)) \\[12pt] = {} & E\left.\begin{cases} \vdots \\ C_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} \\ & {} + \var\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases} \right\} \\[12pt] = {} & \sum_{i=1}^M \alpha_i C_i + \sum_{i=1}^M \alpha_i(\mu_i-\bar\mu)(\mu_i-\bar\mu)^T, \end{align} where $$\bar\mu=\sum_{i=1}^M \alpha_i\mu_i$$.

• Aren't you missing probabilities $\alpha_i$ in the last two summations in your answer? You seem to be using $\alpha_i = M^{-1}$ for all $i$. Commented Sep 15, 2012 at 13:19
• @DilipSarwate : Fixed. Yes, it's weighted average with generally unequal weights. Commented Sep 15, 2012 at 18:56
• @MichaelHardy So, here $var(x)$ is the covariance matrix user is looking at, am I right? Commented May 16, 2015 at 5:26
• Yes. ${}\qquad{}$ Commented May 16, 2015 at 6:45
• @MichaelHardy If you may clear one more doubt of mine regarding covariance matrices, that would be great. Thanks for replying promptly yesterday. Commented May 16, 2015 at 19:57

A mixture model commonly refers to a weighted sum of densities, not a weighted sum of random variables as in Sasha's answer As a simplest example, a (scalar) random variable $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z)=\alpha f_X(z)+(1−\alpha)f_Y(z), ~0<\alpha<1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2)\neq(\mu_Y,\sigma_Y^2)$. It follows straightforwardly that \begin{aligned} E[Z]&= \int_{-\infty}^\infty zf_Z(z)\,\mathrm dz& &= \alpha\mu_X + (1-\alpha)\mu_Y\\ E[Z^2]&= \int_{-\infty}^\infty z^2f_Z(z)\,\mathrm dz& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2)\\ \text{var}(Z) &= E[Z^2] - (E[Z])^2& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2 \end{aligned} which unfortunately does not simplify to a pretty formula.

More generally, a mixture density would have $n, n > 1,$ terms with positive weights $\alpha_1, \alpha_2, \ldots, \alpha_n$ summing to $1$. It is simplest to think of a partition of the sample space into events $A_k, 1 \leq k \leq n$, with $P(A_k) = \alpha_k$. Then, $Z$ is a random variable whose conditional distribution given $A_k$ is a Gaussian distribution $f_k(z) \sim N(\mu_k,\sigma_k^2)$, and thus the unconditional distribution is, via the law of total probability, $$f(z) = \sum_{k=1}^n \alpha_k f_k(z).$$ The ungainly expression $\alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2$ derived earlier for the variance of $Z$ can thus be manipulated into $$\bigr[\alpha\sigma_X^2 + (1-\alpha)\sigma_Y^2\bigr] + \biggr(\bigr[\alpha\mu_X^2 + (1-\alpha)\mu_Y^2\bigr] - \bigr[\alpha\mu_X + (1-\alpha)\mu_Y\bigr]^2\biggr)$$ which, while still not pretty, can be identified as an illustration of the conditional variance formula:

The variance of $Z$ is the mean of the conditional variance plus the variance of the conditional mean.

When $Z$ is a vector random variable whose conditional distributions are jointly Gaussian with mean vector $\mu$ and covariance matrix $C_i$, similar calculations can be done, and the unconditional covariance matrix computed. Suppose that the conditional density of $Z$ given $A_k$ is a jointly Gaussian density with mean vector $\mu^{(k)}$ and covariance matrix $C^{(k)}$. Then, $$E[Z_i] = \sum_{k=1}^n \alpha_k \mu_i^{(k)} ~\text{and}~ E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$$ giving \begin{align}C_{i,j} = \text{cov}(Z_i,Z_j) &= \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr) - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\\ &= \left(\sum_{k=1}^n \alpha_k C_{i,j}^{(k)}\right) + \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\mu_j^{(k)} - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\right) \end{align} which is an illustration of the conditional covariance formula:

The covariance of two random variables is the mean of the conditional covariances plus the covariance of the conditional means.

• Could you explain how you get $E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$ ? Thank you
– Pop
Commented Jul 12, 2013 at 9:21
• @Pop $E[Z_i Z_j] = E_A[ E[ Z_i Z_j | A ] ] = E_A[ Cov[Z_i|A, Z_j|A] + E[Z_i|A] E[Z_j|A] ]$ Commented Jul 28, 2014 at 12:36

Motivated by the answer provided by Michael Hardy, a formal solution to such question might be formulated as follows:

By introducing a new hidden variable $$I$$ to represent the identity of the local model, the probability of the Gaussian mixtures can be decomposed as: $$p(x|I=i)=\mathcal{N}(\mu_i,C_i)$$ $$p(I=i)=\alpha_i$$ Therefore, $$E(x)=E[E(x|I=i)]=\sum_{i=1}^{M} \alpha_i \mu_i$$ \begin{align} Var(x)&=E[Var(x|I=i)]+Var[E(x|I=i)] \\ &=\sum_{i=1}^{M} \alpha_i C_i + \sum_{i=1}^{M} \alpha_i (\mu_i-\bar{\mu})(\mu_i-\bar{\mu})^T \end{align} where $$\bar{\mu}=E(x)$$. Moreover, the Law of total expectation and Law of total variance have been used in above two equations.