Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I have a Gaussian mixture model, given by: $$ x \sim \sum_{i = 1}^M \alpha_i N(\mu_i, C_i) $$

Is there a way I can compute the overall covariance matrix if $x$? I would like to say "$x$ has a covariance matrix given by $C$".

share|improve this question
1  
Is $x$ a vector, or a scalar random variable? –  Sasha Sep 14 '12 at 21:50
    
Please avoid using acronyms. Using expanded names allows for ease of searching, and invites those uninitiated to look the term up and learn about it. –  Sasha Sep 14 '12 at 21:59
    
Removed the tag gauss-sum as that refers to a different concept. –  Sasha Sep 14 '12 at 22:00
    
My answer is at least simpler than the others. But also, I think it's the more-or-less natural way to do it and uses a technique that ought to become habitual for anyone who works with expectations and variances. –  Michael Hardy Sep 15 '12 at 2:13
    
@shasha : vectors.. sorry about that –  NSR Sep 16 '12 at 1:42
add comment

3 Answers

up vote 3 down vote accepted

$\newcommand{\var}{\operatorname{var}}$ You can write $x = y + \text{error}$, where $y = \mu_i$ with probability $\alpha_i$, for $i=1,\ldots,M$, and the conditional probability distribution of the "error" given $y$ is $N(0,C_i)$. Then we have $$ E(x) = E(E(x\mid y)) = E\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} = \sum_{i=1}^M\alpha_i\mu_i, $$ and $$ \begin{align} \var(x) & = E(\var(x\mid y)) + \var(E(x \mid y)) \\[12pt] & = E\left.\begin{cases} \vdots \\ C_i & \text{with probability }\alpha_i \\ \vdots \end{cases}\right\} + \var\left.\begin{cases} \vdots \\ \mu_i & \text{with probability }\alpha_i \\ \vdots \end{cases} \right\} \\[12pt] & = \sum_{i=1}^M \alpha_i C_i + \sum_{i=1}^M \alpha_i(\mu_i-\bar\mu)(\mu_i-\bar\mu)^T, \end{align} $$ where $\bar\mu=\sum_{i=1}^M \alpha_i\mu_i$.

share|improve this answer
    
Aren't you missing probabilities $\alpha_i$ in the last two summations in your answer? You seem to be using $\alpha_i = M^{-1}$ for all $i$. –  Dilip Sarwate Sep 15 '12 at 13:19
    
@DilipSarwate : Fixed. Yes, it's weighted average with generally unequal weights. –  Michael Hardy Sep 15 '12 at 18:56
add comment

I would assume that you talking about linear combination of independent Gaussian random vectors.

Let $X_i$ be a random Gaussian vector with mean $\mu_i$ and covariance matrix $C_i$: $$ \mathbb{E}\left(X_i\right) = \mu_i, \quad \mathbb{E}\left((X_i-\mu_i) \otimes (X_i-\mu_i)\right) = C_i $$ Let $Z = \sum_{i=1}^n \alpha_i X_i$, where $\alpha_i \in \mathbb{R}$. Clearly $Z$ is a Gaussian random vector, as a linear combination of Gaussian random vectors. It is thus determined by its mean vector, and covariance matrix.

Mean vector is easy to come by, using linearity of the expectation: $$ \mathbb{E}\left(Z\right) = \mathbb{E}\left(\sum_{i=1}^n \alpha_i X_i\right) = \sum_{i=1}^n \alpha_i \mathbb{E}\left(X_i\right) = \sum_{i=1}^n \alpha_i \mu_i $$ The covariance matrix, using linearity and independence of $X_i$ and $X_j$ for $i\not=j$: $$ \mathbb{Cov}\left(Z, Z\right) = \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j \mathbb{Cov}\left(X_i, X_j \right) \stackrel{\text{indep.}}{=} \sum_{i=1}^n \alpha_i^2 \mathbb{Cov}\left(X_i, X_i \right) = \sum_{i=1}^n \alpha_i^2 C_i $$

share|improve this answer
3  
A mixture model commonly refers to a weighted sum of densities, not variables. As a simplest example, $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z) = \alpha f_X(z) + (1-\alpha)f_Y(z), ~0 < \alpha < 1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2) \neq (\mu_Y,\sigma_Y^2)$ –  Dilip Sarwate Sep 14 '12 at 22:26
    
Thank you, Dilip. I did not know. In that case the question is really trivial, probably a homework. –  Sasha Sep 14 '12 at 22:39
add comment

A mixture model commonly refers to a weighted sum of densities, not a weighted sum of random variables as in Sasha's answer As a simplest example, a (scalar) random variable $Z$ is said to have a mixture Gaussian density if its probability density function is $$f_Z(z)=\alpha f_X(z)+(1−\alpha)f_Y(z), ~0<\alpha<1$$ where $X$ and $Y$ are Gaussian random variables with different densities, that is, $(\mu_X,\sigma_X^2)\neq(\mu_Y,\sigma_Y^2)$. It follows straightforwardly that $$\begin{aligned} E[Z]&= \int_{-\infty}^\infty zf_Z(z)\,\mathrm dz& &= \alpha\mu_X + (1-\alpha)\mu_Y\\ E[Z^2]&= \int_{-\infty}^\infty z^2f_Z(z)\,\mathrm dz& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2)\\ \text{var}(Z) &= E[Z^2] - (E[Z])^2& &= \alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2 \end{aligned}$$ which unfortunately does not simplify to a pretty formula.

More generally, a mixture density would have $n, n > 1,$ terms with positive weights $\alpha_1, \alpha_2, \ldots, \alpha_n$ summing to $1$. It is simplest to think of a partition of the sample space into events $A_k, 1 \leq k \leq n$, with $P(A_k) = \alpha_k$. Then, $Z$ is a random variable whose conditional distribution given $A_k$ is a Gaussian distribution $f_k(z) \sim N(\mu_k,\sigma_k^2)$, and thus the unconditional distribution is, via the law of total probability, $$f(z) = \sum_{k=1}^n \alpha_k f_k(z).$$ The ungainly expression $\alpha(\sigma_X^2+\mu_X^2) + (1-\alpha)(\sigma_Y^2 + \mu_Y^2) - [\alpha\mu_X + (1-\alpha)\mu_Y]^2$ derived earlier for the variance of $Z$ can thus be manipulated into $$\bigr[\alpha\sigma_X^2 + (1-\alpha)\sigma_Y^2\bigr] + \biggr(\bigr[\alpha\mu_X^2 + (1-\alpha)\mu_Y^2\bigr] - \bigr[\alpha\mu_X + (1-\alpha)\mu_Y\bigr]^2\biggr)$$ which, while still not pretty, can be identified as an illustration of the conditional variance formula:

The variance of $Z$ is the mean of the conditional variance plus the variance of the conditional mean.

When $Z$ is a vector random variable whose conditional distributions are jointly Gaussian with mean vector $\mu$ and covariance matrix $C_i$, similar calculations can be done, and the unconditional covariance matrix computed. Suppose that the conditional density of $Z$ given $A_k$ is a jointly Gaussian density with mean vector $\mu^{(k)}$ and covariance matrix $C^{(k)}$. Then, $$E[Z_i] = \sum_{k=1}^n \alpha_k \mu_i^{(k)} ~\text{and}~ E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$$ giving $$\begin{align}C_{i,j} = \text{cov}(Z_i,Z_j) &= \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr) - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\\ &= \left(\sum_{k=1}^n \alpha_k C_{i,j}^{(k)}\right) + \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\mu_j^{(k)} - \left(\sum_{k=1}^n \alpha_k \mu_i^{(k)}\right) \left(\sum_{k=1}^n \alpha_k \mu_j^{(k)}\right)\right) \end{align}$$ which is an illustration of the conditional covariance formula:

The covariance of two random variables is the mean of the conditional covariances plus the covariance of the conditional means.

share|improve this answer
    
Could you explain how you get $E[Z_iZ_j] = \sum_{k=1}^n \alpha_k \bigr(C_{i,j}^{(k)} + \mu_i^{(k)}\mu_j^{(k)}\bigr)$ ? Thank you –  Pop Jul 12 '13 at 9:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.