Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

If a PDF is re-constructed as following $p(x)=\sum_{k=1}^{K}\pi_k p(x|k),$ where $p(x|k)$ has mean $\mu_k$ and covariance matrix $\Sigma_k$ , compute the mean and covariance of the new mixture distribution.

Finding the mean is trivial , so say that I've found the mean and denote it as $mean=\left[ \begin{array}{cl} mean_{1} \\ ... \\ mean_{D} \end{array}\right]$

I'm quite confused about the dimension of covariance matrix, say I have $D$ dimension for each observation $x=\left[ \begin{array}{cl} x_{1} \\ ... \\ x_{D} \end{array}\right]$, then dimension of covariance matris should be $D \times D,$ but what 's the concrete element in position $<i,j>$ ?

share|improve this question
    
By far what I got is $\Sigma=(\sum_{k=1}^{K} \pi_k \Sigma_k) - mean \cdot mean^T + [\sum_{k=1}^K \pi_k \mu_{ki} \mu_{kj}]$, where (i,j) means the position in a matrix –  genxium May 20 '12 at 19:58

1 Answer 1

I think what you mean is this. There is a random variable $Y$ with possible values $1, \ldots, K$ and corresponding probabilities $\pi_k$. Your random variable $X$ is a $d$-dimensional vector with conditional expectations $\text{E}[X | Y=k] = \mu_k$ (a $d$-dimensional vector) and conditional covariances $\text{Cov}(X | Y=k) = C_k$ (a $d \times d$ matrix). I'd rather use this instead of $\Sigma_k$ because I want to use $\sum_k$ for summation over $k$. Thus $\text{Cov}(X_i,X_j | Y = k) = \text{E}(X_i X_j | Y = k) - \text{E}(X_i | Y=k) \text{E}(X_j | Y = k) = (C_k)_{ij}$. Now the unconditional expectation is $\text{E}(X) = \sum_k E(X| Y=k) \pi_k = \sum_k \mu_k \pi_k$. On the other hand, $$\text{E}(X_i X_j) = \sum_k \text{E}(X_i X_j | Y=k) \pi_k = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j\right) \pi_k$$ so $$ \eqalign{\text{Cov}(X_i,X_j) &= \text{E}(X_i X_j) - E(X_i) E(X_j)\cr & = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j \right) \pi_k - \sum_k (\mu_k)_i \pi_k \sum_{k'} (\mu_k')_j \pi_{k'}\cr} $$ which can be expressed as $$ \text{Cov}(X_i,X_j) = \text{E}\left[{\text{Cov}(X_i,X_j | Y)}\right] + \text{Cov}(E(X_i|Y), E(X_j | Y))$$ the "law of total covariance" formula: see http://en.wikipedia.org/wiki/Law_of_total_covariance These entries form a matrix $$ C = \sum_k C_k \pi_k + \sum_k \mu_k^T \mu_k \pi_k - \mu^T \mu $$ where $\mu = E(X)$.

share|improve this answer
    
Thanks for you answer buddy, but I can't tell if this is right or wrong because I actually got the same reasoning, as my exams are just finished yesterday, I'll consult my professor for this problem soon, hope he will give me the same result :) –  genxium May 26 '12 at 18:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.