# covariance of Gaussian Mixtures

I have a Gaussian mixture model, given by: $$x \sim \sum_{i = 1}^M \alpha_i N(\mu_i, C_i)$$

Is there a way I can compute the overall covariance matrix if $x$? I would like to say "$x$ has a covariance matrix given by $C$".

-
Is $x$ a vector, or a scalar random variable? –  Sasha Sep 14 '12 at 21:50
Please avoid using acronyms. Using expanded names allows for ease of searching, and invites those uninitiated to look the term up and learn about it. –  Sasha Sep 14 '12 at 21:59
Removed the tag gauss-sum as that refers to a different concept. –  Sasha Sep 14 '12 at 22:00
My answer is at least simpler than the others. But also, I think it's the more-or-less natural way to do it and uses a technique that ought to become habitual for anyone who works with expectations and variances. –  Michael Hardy Sep 15 '12 at 2:13
@shasha : vectors.. sorry about that –  NSR Sep 16 '12 at 1:42

$\newcommand{\var}{\operatorname{var}}$ You can write $x = y + \text{error}$, where $y = \mu_i$ with probability $\alpha_i$, for $i=1,\ldots,M$, and the conditional probability distribution of the "error" given $y$ is $N(0,C_i)$. Then we have $$E(x) = E(E(x\mid y)) = E\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} = \sum_{i=1}^M\alpha_i\mu_i,$$ and \begin{align} \var(x) & = E(\var(x\mid y)) + \var(E(x \mid y)) \\[12pt] & = E\left.\begin{cases} \vdots \\ C_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} + \var\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases} \right\} \\[12pt] & = \sum_{i=1}^M \alpha_i C_i + \sum_{i=1}^M \alpha_i(\mu_i-\bar\mu)(\mu_i-\bar\mu)^T, \end{align} where $\bar\mu=\sum_{i=1}^M \alpha_i\mu_i$.

-
Aren't you missing probabilities $\alpha_i$ in the last two summations in your answer? You seem to be using $\alpha_i = M^{-1}$ for all $i$. –  Dilip Sarwate Sep 15 '12 at 13:19
@DilipSarwate : Fixed. Yes, it's weighted average with generally unequal weights. –  Michael Hardy Sep 15 '12 at 18:56
@MichaelHardy So, here $var(x)$ is the covariance matrix user is looking at, am I right? –  Parag S. Chandakkar May 16 at 5:26
Yes. ${}\qquad{}$ –  Michael Hardy May 16 at 6:45
@MichaelHardy If you may clear one more doubt of mine regarding covariance matrices, that would be great. Thanks for replying promptly yesterday. –  Parag S. Chandakkar May 16 at 19:57

A mixture model commonly refers to a weighted sum of densities, not a weighted sum of random variables as in Sasha's answer As a simplest example, a (scalar) random variable $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z)=\alpha f_X(z)+(1−\alpha)f_Y(z), ~0<\alpha<1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2)\neq(\mu_Y,\sigma_Y^2)$. It follows straightforwardly that \begin{aligned} E[Z]&= \int_{-\infty}^\infty zf_Z(z)\,\mathrm dz& &= \alpha\mu_X + (1-\alpha)\mu_Y\\ E[Z^2]&= \int_{-\infty}^\infty z^2f_Z(z)\,\mathrm dz& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2)\\ \text{var}(Z) &= E[Z^2] - (E[Z])^2& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2 \end{aligned} which unfortunately does not simplify to a pretty formula.

More generally, a mixture density would have $n, n > 1,$ terms with positive weights $\alpha_1, \alpha_2, \ldots, \alpha_n$ summing to $1$. It is simplest to think of a partition of the sample space into events $A_k, 1 \leq k \leq n$, with $P(A_k) = \alpha_k$. Then, $Z$ is a random variable whose conditional distribution given $A_k$ is a Gaussian distribution $f_k(z) \sim N(\mu_k,\sigma_k^2)$, and thus the unconditional distribution is, via the law of total probability, $$f(z) = \sum_{k=1}^n \alpha_k f_k(z).$$ The ungainly expression $\alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2$ derived earlier for the variance of $Z$ can thus be manipulated into $$\bigr[\alpha\sigma_X^2 + (1-\alpha)\sigma_Y^2\bigr] + \biggr(\bigr[\alpha\mu_X^2 + (1-\alpha)\mu_Y^2\bigr] - \bigr[\alpha\mu_X + (1-\alpha)\mu_Y\bigr]^2\biggr)$$ which, while still not pretty, can be identified as an illustration of the conditional variance formula:

The variance of $Z$ is the mean of the conditional variance plus the variance of the conditional mean.

When $Z$ is a vector random variable whose conditional distributions are jointly Gaussian with mean vector $\mu$ and covariance matrix $C_i$, similar calculations can be done, and the unconditional covariance matrix computed. Suppose that the conditional density of $Z$ given $A_k$ is a jointly Gaussian density with mean vector $\mu^{(k)}$ and covariance matrix $C^{(k)}$. Then, $$E[Z_i] = \sum_{k=1}^n \alpha_k \mu_i^{(k)} ~\text{and}~ E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$$ giving \begin{align}C_{i,j} = \text{cov}(Z_i,Z_j) &= \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr) - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\\ &= \left(\sum_{k=1}^n \alpha_k C_{i,j}^{(k)}\right) + \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\mu_j^{(k)} - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\right) \end{align} which is an illustration of the conditional covariance formula:

The covariance of two random variables is the mean of the conditional covariances plus the covariance of the conditional means.

-
Could you explain how you get $E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$ ? Thank you –  Pop Jul 12 '13 at 9:21
@Pop $E[Z_i Z_j] = E_A[ E[ Z_i Z_j | A ] ] = E_A[ Cov[Z_i|A, Z_j|A] + E[Z_i|A] E[Z_j|A] ]$ –  tflutre Jul 28 '14 at 12:36

I would assume that you talking about linear combination of independent Gaussian random vectors.

Let $X_i$ be a random Gaussian vector with mean $\mu_i$ and covariance matrix $C_i$: $$\mathbb{E}\left(X_i\right) = \mu_i, \quad \mathbb{E}\left((X_i-\mu_i) \otimes (X_i-\mu_i)\right) = C_i$$ Let $Z = \sum_{i=1}^n \alpha_i X_i$, where $\alpha_i \in \mathbb{R}$. Clearly $Z$ is a Gaussian random vector, as a linear combination of Gaussian random vectors. It is thus determined by its mean vector, and covariance matrix.

Mean vector is easy to come by, using linearity of the expectation: $$\mathbb{E}\left(Z\right) = \mathbb{E}\left(\sum_{i=1}^n \alpha_i X_i\right) = \sum_{i=1}^n \alpha_i \mathbb{E}\left(X_i\right) = \sum_{i=1}^n \alpha_i \mu_i$$ The covariance matrix, using linearity and independence of $X_i$ and $X_j$ for $i\not=j$: $$\mathbb{Cov}\left(Z, Z\right) = \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j \mathbb{Cov}\left(X_i, X_j \right) \stackrel{\text{indep.}}{=} \sum_{i=1}^n \alpha_i^2 \mathbb{Cov}\left(X_i, X_i \right) = \sum_{i=1}^n \alpha_i^2 C_i$$

-
A mixture model commonly refers to a weighted sum of densities, not variables. As a simplest example, $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z) = \alpha f_X(z) + (1-\alpha)f_Y(z), ~0 < \alpha < 1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2) \neq (\mu_Y,\sigma_Y^2)$ –  Dilip Sarwate Sep 14 '12 at 22:26
Thank you, Dilip. I did not know. In that case the question is really trivial, probably a homework. –  Sasha Sep 14 '12 at 22:39