I have very limited knowledge of linear algebra and therefore I don't have an geometrical intuition behind PCA.

Why the eigen vectors (which are simply defined as vectors whose direction doesn't change after a linear transformation) are also the directions that maximize variance? I have seen PCA definition using Lagrange multipliers but I would like to have the geometrical intuition behind it.



You're asking for intuition around the following fact: "If $A$ is the covariance matrix, and you want to maximize (or minimize) $f(x)=x^TAx$ under the constraint that $\|x\|^2=1$, then any solution $x_0$ will be such that $x_0$ and $Ax_0$ are collinear."

To see this geometrically, consider the level sets of $x^TAx$. Those are the set of point where $x^TAx$ assumes the same value. You can verify that those level sets are ellipsoids whose axes are aligned along $A$'s eigenvectors.

Values of $x^TAx

Now let's go back to the maximization of $x^TAx$ under the constraint that $\|x\|^2=1$. To build an intuition, assume we take a gradient descent approach. The way it works: You start somewhere, then start following the gradient of the function $x^TAx$, while continuing to satisfy the constraint $\|x\|^2=1$, and repeat until you can no longer move.

Now, note the following

  • The gradient of $x^TAx$ is equal to $Ax$.

  • If you are at a point $x$ satisfying the constraint, and you want to move away while still satisfying that constraint, then you must move in a direction that's orthogonal to the direction of the gradient of that constraint. In the case of the $\|x\|^2=1$ constraint, that means you must move in a direction orthogonal to $x$.

In the picture below, we represent the level sets of the function $x^TAx$ to optimize. Suppose you are at the point where the red and green arrows meet. The red arrow is $Ax$, and the green arrow is $x$. Then you can easily see that by moving in a direction orthogonal to the green arrow, you can reach another level set, one with a higher value of $x^TAx$ (as indicated by the red arrow). So as long as you're in such a configuration where $Ax$ and $x$ are not collinear, you can always move to another point where the constraint is satisfied and you increase the function $x^TAx$.

Contrast this with the situation where both $Ax$ (pink arrow) and $x$ (blue arrow) are collinear. Now, if you move in a direction orthogonal to $x$ (to satisfy the constraint), you'll also move in a direction that's orthogonal to $Ax$, achieving no variation of the function to optimize. In that case, you're already at an extremum.

This proves that extrema of a function under a constraint are achieved when both gradients of the function and constraint are collinear.

Level sets and gradients

Finally, note that none of this depends on the particular form for the function as $f(x)=x^TAx$ or the constraint as $\|x\|^2=1$. The reasoning I presented is a geometric proof of Lagrange multipliers for general functions and constraints.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.