If a PDF is re-constructed as following $p(x)=\sum_{k=1}^{K}\pi_k p(x|k),$ where $p(x|k)$ has mean $\mu_k$ and covariance matrix $\Sigma_k$ , compute the mean and covariance of the new mixture distribution.

Finding the mean is trivial , so say that I've found the mean and denote it as $mean=\left[ \begin{array}{cl} mean_{1} \\ ... \\ mean_{D} \end{array}\right]$

I'm quite confused about the dimension of covariance matrix, say I have $D$ dimension for each observation $x=\left[ \begin{array}{cl} x_{1} \\ ... \\ x_{D} \end{array}\right]$, then dimension of covariance matris should be $D \times D,$ but what 's the concrete element in position $<i,j>$ ?

  • $\begingroup$ By far what I got is $\Sigma=(\sum_{k=1}^{K} \pi_k \Sigma_k) - mean \cdot mean^T + [\sum_{k=1}^K \pi_k \mu_{ki} \mu_{kj}]$, where (i,j) means the position in a matrix $\endgroup$ – genxium May 20 '12 at 19:58
  • $\begingroup$ Basically the same question (and answer) here: math.stackexchange.com/questions/195911/… $\endgroup$ – Yibo Yang Mar 4 at 6:20

I think what you mean is this. There is a random variable $Y$ with possible values $1, \ldots, K$ and corresponding probabilities $\pi_k$. Your random variable $X$ is a $d$-dimensional vector with conditional expectations $\text{E}[X | Y=k] = \mu_k$ (a $d$-dimensional vector) and conditional covariances $\text{Cov}(X | Y=k) = C_k$ (a $d \times d$ matrix). I'd rather use this instead of $\Sigma_k$ because I want to use $\sum_k$ for summation over $k$. Thus $\text{Cov}(X_i,X_j | Y = k) = \text{E}(X_i X_j | Y = k) - \text{E}(X_i | Y=k) \text{E}(X_j | Y = k) = (C_k)_{ij}$. Now the unconditional expectation is $\text{E}(X) = \sum_k E(X| Y=k) \pi_k = \sum_k \mu_k \pi_k$. On the other hand, $$\text{E}(X_i X_j) = \sum_k \text{E}(X_i X_j | Y=k) \pi_k = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j\right) \pi_k$$ so $$ \eqalign{\text{Cov}(X_i,X_j) &= \text{E}(X_i X_j) - E(X_i) E(X_j)\cr & = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j \right) \pi_k - \sum_k (\mu_k)_i \pi_k \sum_{k'} (\mu_k')_j \pi_{k'}\cr} $$ which can be expressed as $$ \text{Cov}(X_i,X_j) = \text{E}\left[{\text{Cov}(X_i,X_j | Y)}\right] + \text{Cov}(E(X_i|Y), E(X_j | Y))$$ the "law of total covariance" formula: see http://en.wikipedia.org/wiki/Law_of_total_covariance These entries form a matrix $$ C = \sum_k C_k \pi_k + \sum_k \mu_k^T \mu_k \pi_k - \mu^T \mu $$ where $\mu = E(X)$.

  • $\begingroup$ Thanks for you answer buddy, but I can't tell if this is right or wrong because I actually got the same reasoning, as my exams are just finished yesterday, I'll consult my professor for this problem soon, hope he will give me the same result :) $\endgroup$ – genxium May 26 '12 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.