I have troubles understanding a geometrical meaning of a covariance matrix.

Let's say we have a data set containing two points (-1,1), (-1,2) and write them in to the matrix

$$D = \begin{bmatrix} -1 & -1\\ 1 & 2 \end{bmatrix}. $$

Then we'll choose the unit-vector $v = \left( \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}} \right)^T$.

And now, in this article is the following statement:

So, if we would like to represent the covariance matrix with a vector and its magnitude, we should simply try to find the vector that points into the direction of the largest spread of the data, and whose magnitude equals the spread (variance) in this direction.

If we define this vector as $v$, then the projection of our data $D$ onto this vector is obtained as $v^TD$, and the variance of the projected data is $v^T\Sigma v$.

The covariance matrix $\Sigma$ looks like this:

$$ \Sigma = \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix} $$

Also, the projection of $D$ onto $v$ is the following vector:

$$ v^TD = \begin{bmatrix} 0\\ \sqrt{2} - \frac{1}{\sqrt{2}} \end{bmatrix} $$

So, in our case

$$ v^T\Sigma v = \left[ \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}} \right] \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix} \begin{bmatrix} \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} \end{bmatrix} = \frac{1}{4}. $$

So, could you, please, explain, how can $v^T\Sigma v$ be a variance of values projected on $v$? I would really appreciate "the proof", that $v^T\Sigma v$ equals variance, as I don't see, why it is true.

  • 1
    $\begingroup$ The variance is $v^T\Sigma{v}$ not $v^TDv$ $\endgroup$
    – David M.
    Mar 18 '19 at 1:18
  • $\begingroup$ @DavidM. Thank you for the correction! $\endgroup$
    – Eenoku
    Mar 18 '19 at 1:45
  • 1
    $\begingroup$ Also, $v^TD\neq Dv$, as $D$ is not symmetric! $\endgroup$
    – David M.
    Mar 18 '19 at 1:49
  • $\begingroup$ @DavidM. Thank you again! $\endgroup$
    – Eenoku
    Mar 18 '19 at 1:59
  • 1
    $\begingroup$ @DavidM. They weren't pedantic, but necessary :) Something like $\begin{bmatrix} 0&0\\0&\frac{1}{2} \end{bmatrix}$ ? I'm afraid I exchanged columns with rows before in hurry... $\endgroup$
    – Eenoku
    Mar 18 '19 at 2:20

Your original data matrix containing the points (in this case, 2-dimension) is

$$ D = \begin{bmatrix} -1 & -1\\ 1 & 2 \end{bmatrix} $$

A shift of origin can be performed to center the points (-1,1) and (-1,2).
The mean of the x coordinates = -1.
The mean of the y coordinates = 3/2.

This will transform the data matrix to

$$ D = \begin{bmatrix} -1-(-1) & -1-(-1)\\ 1-\frac{3}{2} & 2-\frac{3}{2} \end{bmatrix} = \begin{bmatrix} 0 & 0\\ -\frac{1}{2} & \frac{1}{2} \end{bmatrix}$$

The covariance matrix is given by $DD^T$

$$ DD^T = \Sigma = \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix} $$

The projection (in this case, into 1-dimension) of the points in $D$ on a vector $v$ is given by $v^TD$.

$v^TD$ can now be thought of as your 'Projected Data Matrix (P)' whose components give the coordinates of the points in the projected space (in this case, along the vector $v$).

Noting that $DD^T=\Sigma$, the covariance (variance, if 1-dimension) of $P$ is thus given by $PP^T$,

$$PP^T=v^TD(v^TD)^T=v^TDD^Tv=v^T\Sigma v$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.