Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If a PDF is re-constructed as following $p(x)=\sum_{k=1}^{K}\pi_k p(x|k),$ where $p(x|k)$ has mean $\mu_k$ and covariance matrix $\Sigma_k$ , compute the mean and covariance of the new mixture distribution.

Finding the mean is trivial , so say that I've found the mean and denote it as $mean=\left[ \begin{array}{cl} mean_{1} \\ ... \\ mean_{D} \end{array}\right]$

I'm quite confused about the dimension of covariance matrix, say I have $D$ dimension for each observation $x=\left[ \begin{array}{cl} x_{1} \\ ... \\ x_{D} \end{array}\right]$, then dimension of covariance matris should be $D \times D,$ but what 's the concrete element in position $<i,j>$ ?

share|cite|improve this question
By far what I got is $\Sigma=(\sum_{k=1}^{K} \pi_k \Sigma_k) - mean \cdot mean^T + [\sum_{k=1}^K \pi_k \mu_{ki} \mu_{kj}]$, where (i,j) means the position in a matrix – genxium May 20 '12 at 19:58

I think what you mean is this. There is a random variable $Y$ with possible values $1, \ldots, K$ and corresponding probabilities $\pi_k$. Your random variable $X$ is a $d$-dimensional vector with conditional expectations $\text{E}[X | Y=k] = \mu_k$ (a $d$-dimensional vector) and conditional covariances $\text{Cov}(X | Y=k) = C_k$ (a $d \times d$ matrix). I'd rather use this instead of $\Sigma_k$ because I want to use $\sum_k$ for summation over $k$. Thus $\text{Cov}(X_i,X_j | Y = k) = \text{E}(X_i X_j | Y = k) - \text{E}(X_i | Y=k) \text{E}(X_j | Y = k) = (C_k)_{ij}$. Now the unconditional expectation is $\text{E}(X) = \sum_k E(X| Y=k) \pi_k = \sum_k \mu_k \pi_k$. On the other hand, $$\text{E}(X_i X_j) = \sum_k \text{E}(X_i X_j | Y=k) \pi_k = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j\right) \pi_k$$ so $$ \eqalign{\text{Cov}(X_i,X_j) &= \text{E}(X_i X_j) - E(X_i) E(X_j)\cr & = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j \right) \pi_k - \sum_k (\mu_k)_i \pi_k \sum_{k'} (\mu_k')_j \pi_{k'}\cr} $$ which can be expressed as $$ \text{Cov}(X_i,X_j) = \text{E}\left[{\text{Cov}(X_i,X_j | Y)}\right] + \text{Cov}(E(X_i|Y), E(X_j | Y))$$ the "law of total covariance" formula: see These entries form a matrix $$ C = \sum_k C_k \pi_k + \sum_k \mu_k^T \mu_k \pi_k - \mu^T \mu $$ where $\mu = E(X)$.

share|cite|improve this answer
Thanks for you answer buddy, but I can't tell if this is right or wrong because I actually got the same reasoning, as my exams are just finished yesterday, I'll consult my professor for this problem soon, hope he will give me the same result :) – genxium May 26 '12 at 18:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.