5
$\begingroup$

My understanding is that given matrix X, I can find its corresponding covariance matrix by:

  1. finding the means of each column
  2. subtracting each mean from each value in its respective column and
  3. multiplying the resulting matrix by its own transpose. Let's call this matrix C.

Here is what it would look like in Python: Y = X - numpy.mean(X, axis = 0) C = numpy.dot(Y, Y.T)

If I do this, I can prove mathematically (and experimentally using some simple Python code) that det(C) = 0 always.

However, a colleague tells me that using the inverse of a covariance matrix is common in his field and he showed me some R code to demonstrate.

> det(cov(swiss)) [1] 244394171542

I notice that R has several ways of calculating the covariance matrix that leads to different results. I also notice from Googling that some people say the covariance matrix is always singular (eg here) whereas others say it is not.

So, my question is: why the differences of opinion and what's the true answer?

EDIT: I discovered that the determinant is only zero if the matrix is square. If anybody knows the proof for this or can throw some further light on the matter, I'd be grateful.

$\endgroup$
  • 1
    $\begingroup$ Are you sure your colleague has the same thing in mind when he's talking about the covariance matrix? $\endgroup$ – Arnaud D. Jan 28 '18 at 11:38
1
$\begingroup$

No, the covariance matrix is not always singular. Counterexample: \begin{align} X&=(\mathbf x_1,\mathbf x_2,\mathbf x_3)=\pmatrix{1&2&3\\ 1&1&4},\\ \bar{\mathbf x}&=\frac13(\mathbf x_1+\mathbf x_2+\mathbf x_3)=\pmatrix{2\\ 2},\\ Y&=(\mathbf x_1-\bar{\mathbf x},\ \mathbf x_2-\bar{\mathbf x},\ \mathbf x_3-\bar{\mathbf x})=\pmatrix{-1&0&1\\ -1&-1&2},\\ YY^T&=\pmatrix{2&3\\ 3&6},\text{ which is nonsingular}. \end{align} It is true, however, that when the data matrix $X$ is square or "tall", i.e. when $X$ is $n\times m$ with $n\ge m$ (or equivalently, when the number of data points $m$ does not exceed the number of variables $n$), $YY^T$ is always singular. This is evident because by definition, the sum of all columns of $Y$ is the zero vector. Hence $Y$ has deficient column rank and in turn, $\operatorname{rank}(YY^T)=\operatorname{rank}(Y)<m\le n$.

$\endgroup$
0
$\begingroup$

Yes your statement is true. Let $X$ a matrix $n\times m$. If you do the algorithm that you wrote above (until the step $2$) you obtain a matrix $Z$ that have the sum of each column equal to $0$.

$$\forall j=1,\cdots,m \quad x_{1,j}-\sum_{i=1}^n\frac{x_{i,j}}{n}+\cdots+ x_{n,j}-\sum_{i=1}^n\frac{x_{i,j}}{n}=\sum_{i=1}^nx_{i,j}-n\sum_{i=1}^n\frac{x_{i,j}}{n}=0$$

Now if you do the product $Y=Z \cdot Z^T$ this statement continues to be true, and such that the matrix $Y$ is symmetric the statement is true for each row too. For example for the first column:

$$y_{1,1}=\sum_{i=1}^m z_{1,i}^2$$ $$y_{2,1}=\sum_{i=1}^mz_{2,i}z_{1,i}$$ $$\vdots$$ $$y_{n,1}=\sum_{i=1}^m z_{n,i}z_{1,i}$$ So $$\sum_{j=1}^n y_{j,1}=\sum_{i=1}^m \Big( z_{1,i}^2+z_{2,i}z_{1,i}+ \cdots + z_{n,i}z_{1,i}\Big)=\sum_{i=1}^m z_{1,i}\left( z_{1,i}+z_{2,i}+ \cdots + z_{n,i} \right)=0$$ Similarly for column=$2,3, \cdots, n$

$\endgroup$
  • $\begingroup$ This does not answer the question... The question is: How come that there are covariance matrices that you can invert, yet with this method you can create matrices whose determinant is zero. The right answer would detail the following: There are covariance matrices that cannot be produced this way. $\endgroup$ – zoli Jan 28 '18 at 11:37
-1
$\begingroup$

The result of matrix multiplication is only an approximation of the true covariance matrix and disregards distributions of random variables that are crucial for accurate calcualtion. Hope it helps.

$\endgroup$
  • $\begingroup$ Very vague and best suited as a comment. $\endgroup$ – Jack D'Aurizio Feb 1 '18 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.