Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Given a matrix X whose rows contain observations collected at some locations. Can someone explain how trace minimization of covariance matrix $XX^T$ can lead to orthogonal / mutually independent observations being selected from a larger set. Some mathematical explanation will be helpful.

share|improve this question
    
I don't understand. Could you elaborate or provide some reference about the property? –  leonbloy Jan 9 '13 at 17:11
    
Sorry, I cannot make sense of all this. –  leonbloy Jan 10 '13 at 12:25
    
@leonbloy Can you be specific which part of the question is confusing or unclear. I can elaborate further. –  hauin Jan 10 '13 at 14:42
    
The question statement is not all clear. "Minimize the trace of a covariance matrix". You can't minimize something if you don't specify which are the variables and which are the constraints. –  leonbloy Jan 10 '13 at 14:48
    
I dimly suspect that it could be related to in signal processing is known as the "whitening" property of the optimal predicion error filter (the filter that minimizes the expected squared error of the prediction -which would correspond to the trace of its covariance matrix- gives as result an "white", uncorrelated -i.e., diagonal covariance- output). But that's just a wild guess. Eg: cs.tut.fi/~tabus/course/ASP/LectureNew7.pdf –  leonbloy Jan 10 '13 at 14:53

1 Answer 1

For a random vector $X\in\mathbb{R}^n$ with covariance matrix $\Sigma_X$, we have \begin{align} \mathbb{E}[\|X\|^2]&=\mbox{Tr}(\Sigma_X)\\ &=\sum_i\lambda_i(\Sigma_X)\\ &\ge n\left(\prod_i\lambda_i(\Sigma_X)\right)^{1/n}\\ &=n\det(\Sigma_X)^{1/n}\\ &\ge \frac{n}{2\pi e}e^{2h(X)/n}, \end{align} where the first inequality is AM-GM and the second inequality follows from the fact that differential entropy $h(X)$ is maximized when $X$ is Gaussian.

Your question relates to equality being achieved in the first inequality. This only happens when the spectrum of $\Sigma_X$ is flat i.e. $\Sigma_X=\lambda I$, which implies that the components of $X$ must be uncorrelated (and hence independent, if $X$ is Gaussian).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.