How is it determined that the Covariance matrix has Eigenvectors which are in the direction of largest variation of a data set?

I suppose to derive this you would maybe use regression line for the normalised data set, after finding the regression line we have found the vector of largest variation. From the regression line vector we can find the orthogonal vector.

Are the vectors chosen to be Eigenvectors? How are the Eigen values found? So we choose these vectors in which they are Eigen vectors of a matrix A and this matrix A happens to be the covariance matrix?

Im basically missing the connection between choosing vectors in direction with greatest variance for a dataset and how these were determined to be related to the Covariance matrix. Because if you choose 3 Eigenvectors and find the matrix A essentially it will be random numbers, so does it turn out that these random numbers are covariances and variances?

enter image description here


For starters, notice that "maximum variance" in some direction is the same as "least squared residuals when projecting orthogonally onto a line with that direction". With that insight, a lot of the derivation becomes very similar to the derivation of the OLS formula.

We want to find a vector $\vec{w}$ with $||\vec{w}|| = 1$ such that $||X \vec{w} ||$ is maximized.

We can write that using a Lagrange multiplier as:

$$\vec{w}^T X^T X \vec{w} + \lambda (1 - \vec{w}^T\vec{w})$$

Differentiating w.r.t. $\vec{w}$ and setting the derivative equal to $0$ gives us:

$$2 X^TX \vec{w} - 2 \lambda \vec{w} = 0$$ which can be rearranged as: $$X^T X \vec{w} = \lambda \vec{w}$$

This is exactly the definition of an eigenvector of $X^TX$.

  • $\begingroup$ Interesting thank you! $\endgroup$ Oct 6 '20 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.