I'm reading my linear algebra book and they have an example of how to construct a covariance matrix for some multidimensional data.

enter image description here

The matrix $M$ is obvious. However, I'm assuming the $\hat{X}_k$ is not so obvious. I'm assuming that $\hat{X}_k$ represents the variance of some random variable?

From elementary stats class I understand that $Var(X) = E[(X - \mu)^2]$ which is a formulation for "average distance away from the mean for some random variable $X$". So, I can see that if $X_k$, some random data vector is a random variable, $X_k - M$ will give the distance from mean, but not the "average" distance from the mean ?

I also know that $Cov(X, Y) = E[(X - E[X])(Y - E[Y])]$ How is the matrix $S$ embodying this formulation? I see that $BB^T$ is the product of a $p \times N$ and a $N \times p$ matrix, so I can imagine that is doing the multiplication $(\hat{X}_i - M)(\hat{X_j}- M) \forall i,j$. But, why is it being divided by $\frac{1}{N-1}$ ?

Furthermore, I get that $S$ being positive semidefinite means that all of its eigenvalues are positive. What do the eigenvalues of a covariance matrix represent?

  • 1
    $\begingroup$ $\hat X$ are your sample observations adjusted by the mean of sample. The variance of $X$ is $\frac 1{n-1} \hat X^T\hat X.$ Moving to multiple variables, $B = [\hat X_1, \hat X_2, \cdots \hat X_N]$, and $\frac 1{N-1} B^TB$ will calculate the variance down the main diagonal of M, and the co-variance of $X_i,X_j$ in the off diagonals. Why $N-1$?... I like to think of it that you lose a degree of information in calculating the sample mean. You might want to look up "unbiased estimators" for a more precise explanation. $\endgroup$ – Doug M Feb 14 '18 at 23:16
  • 1
    $\begingroup$ The eigenvectors of the co-variance matrix represent the "principle components" the eigenvalues represent the loading on the respective principle components. $\endgroup$ – Doug M Feb 14 '18 at 23:23
  • $\begingroup$ @DougM I looked up "unbiased estimators" and have a rough understanding of what they mean. They are statistics about the dataset that don't overestimate or underestimate the true statistic (whether that true statistic is mean, median, etc.). I'm also not sure what you mean by losing a degree of freedom $\endgroup$ – Carpetfizz May 14 '18 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.