Covariance Matrix of mixture distribution If a PDF is re-constructed as following $p(x)=\sum_{k=1}^{K}\pi_k p(x|k),$ where $p(x|k)$ has mean $\mu_k$ and covariance matrix $\Sigma_k$ , compute the mean and covariance of the new mixture distribution. 
Finding the mean is trivial , so say that I've found the mean and denote it as $mean=\left[ \begin{array}{cl} mean_{1} \\ ... \\ mean_{D} \end{array}\right]$
I'm quite confused about the dimension of covariance matrix, say I have $D$ dimension for each observation $x=\left[ \begin{array}{cl} x_{1} \\ ... \\ x_{D} \end{array}\right]$, then dimension of covariance matris should be $D \times D,$ but what 's the concrete element in position $<i,j>$ ?
 A: I think what you mean is this.  There is a random variable $Y$ with possible values $1, \ldots, K$ and corresponding probabilities $\pi_k$.  Your random variable $X$ is a $d$-dimensional vector with 
conditional expectations $\text{E}[X | Y=k] = \mu_k$ (a $d$-dimensional vector) and conditional
covariances $\text{Cov}(X | Y=k) = C_k$ (a $d \times d$ matrix). I'd rather use this instead of $\Sigma_k$ because I want to use $\sum_k$ for summation over $k$. Thus $\text{Cov}(X_i,X_j | Y = k) = \text{E}(X_i X_j | Y = k) - \text{E}(X_i | Y=k) \text{E}(X_j | Y = k) = (C_k)_{ij}$.  Now the unconditional expectation is $\text{E}(X) = \sum_k E(X| Y=k) \pi_k = \sum_k \mu_k \pi_k$.  On the other hand, 
$$\text{E}(X_i X_j) =  \sum_k \text{E}(X_i X_j | Y=k) \pi_k = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j\right) \pi_k$$
so 
$$ \eqalign{\text{Cov}(X_i,X_j) &= \text{E}(X_i X_j) - E(X_i) E(X_j)\cr & = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j \right) \pi_k - \sum_k (\mu_k)_i \pi_k \sum_{k'} (\mu_k')_j \pi_{k'}\cr} $$ 
which can be expressed as $$ \text{Cov}(X_i,X_j) = \text{E}\left[{\text{Cov}(X_i,X_j | Y)}\right] + \text{Cov}(E(X_i|Y), E(X_j | Y))$$ 
the "law of total covariance" formula: see http://en.wikipedia.org/wiki/Law_of_total_covariance
These entries form a matrix
$$ C = \sum_k C_k \pi_k + \sum_k \mu_k^T \mu_k \pi_k - \mu^T \mu $$
where $\mu = E(X)$.
