# On SVD, PCA and optimization

Let's say we got a data matrix $$X_{n,p}$$ wich represent $$n$$ person with $$p$$ feature. We can represent each person in an $$R^n$$ space, then we get a data cloud.

One goal of PCA is to find the "best approximation" of the data cloud in a $$k$$ dimensional subspace of $$R^n$$. This best approximation is introduced in various geometric way:

1. We look for a projection $$P_{E_k}$$ such that the sum of the distance between the "projected person" would be maximum.

2. Similar to "least square"method we look for an "orthogonal projection" which would minimize the sum of distance between the points and their projected.

3. We look for the maximum "inertie" or Variance of the projected cloud which geometrically give the total distance between the projected cloud and his bartcenter.

4. Finally a non geometric approach lead us to look for the "best" low rank (rank $$k$$) approximation of $$X$$ (cf.

Eckart-Young theorem. My questions are the following (please don't miss the drawing at the end)

A) Could you prove the equivalence between all theses statement?

B) In 1) we consider all the projection whereas in 2) we focused on orthogonal projection. How is that possible ? How do we finally get that the "best" projection is an orthogonal one?

PS: All my question raised from a really interesting presentation wich suffer to prove nothing: http://bertrand.michel.perso.math.cnrs.fr/Enseignements/5MS04/ACP-AFC-ACM.pdf Could you prove the equivalence of the 4 point of view ?

Thanks

• "Best approximation" is not a very good description as wellness of fit can be defined in very many different ways. Finding the orientations of largest variance is a better description of what PCA and Karhunen-Loeve Transforms do. It is true we get the best fit in a least squares / $L_2$ energy sense, but that is far from the most relevant error measures for many kinds of problems. Aug 23, 2017 at 17:58
• I know that is not optimal but i would like first undrrstand it and after consider better way to do. Aug 26, 2017 at 6:29
• The link to the PDF is broken. Apr 9, 2021 at 11:06