# Covariance matrix and projection

I have troubles understanding a geometrical meaning of a covariance matrix.

Let's say we have a data set containing two points (-1,1), (-1,2) and write them in to the matrix

$$D = \begin{bmatrix} -1 & -1\\ 1 & 2 \end{bmatrix}.$$

Then we'll choose the unit-vector $$v = \left( \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}} \right)^T$$.

So, if we would like to represent the covariance matrix with a vector and its magnitude, we should simply try to find the vector that points into the direction of the largest spread of the data, and whose magnitude equals the spread (variance) in this direction.

If we define this vector as $$v$$, then the projection of our data $$D$$ onto this vector is obtained as $$v^TD$$, and the variance of the projected data is $$v^T\Sigma v$$.

The covariance matrix $$\Sigma$$ looks like this:

$$\Sigma = \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix}$$

Also, the projection of $$D$$ onto $$v$$ is the following vector:

$$v^TD = \begin{bmatrix} 0\\ \sqrt{2} - \frac{1}{\sqrt{2}} \end{bmatrix}$$

So, in our case

$$v^T\Sigma v = \left[ \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}} \right] \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix} \begin{bmatrix} \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} \end{bmatrix} = \frac{1}{4}.$$

So, could you, please, explain, how can $$v^T\Sigma v$$ be a variance of values projected on $$v$$? I would really appreciate "the proof", that $$v^T\Sigma v$$ equals variance, as I don't see, why it is true.

• The variance is $v^T\Sigma{v}$ not $v^TDv$ Mar 18 '19 at 1:18
• @DavidM. Thank you for the correction! Mar 18 '19 at 1:45
• Also, $v^TD\neq Dv$, as $D$ is not symmetric! Mar 18 '19 at 1:49
• @DavidM. Thank you again! Mar 18 '19 at 1:59
• @DavidM. They weren't pedantic, but necessary :) Something like $\begin{bmatrix} 0&0\\0&\frac{1}{2} \end{bmatrix}$ ? I'm afraid I exchanged columns with rows before in hurry... Mar 18 '19 at 2:20

Your original data matrix containing the points (in this case, 2-dimension) is

$$D = \begin{bmatrix} -1 & -1\\ 1 & 2 \end{bmatrix}$$

A shift of origin can be performed to center the points (-1,1) and (-1,2).
The mean of the x coordinates = -1.
The mean of the y coordinates = 3/2.

This will transform the data matrix to

$$D = \begin{bmatrix} -1-(-1) & -1-(-1)\\ 1-\frac{3}{2} & 2-\frac{3}{2} \end{bmatrix} = \begin{bmatrix} 0 & 0\\ -\frac{1}{2} & \frac{1}{2} \end{bmatrix}$$

The covariance matrix is given by $$DD^T$$

$$DD^T = \Sigma = \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix}$$

The projection (in this case, into 1-dimension) of the points in $$D$$ on a vector $$v$$ is given by $$v^TD$$.

$$v^TD$$ can now be thought of as your 'Projected Data Matrix (P)' whose components give the coordinates of the points in the projected space (in this case, along the vector $$v$$).

Noting that $$DD^T=\Sigma$$, the covariance (variance, if 1-dimension) of $$P$$ is thus given by $$PP^T$$,

$$PP^T=v^TD(v^TD)^T=v^TDD^Tv=v^T\Sigma v$$