2
$\begingroup$

Let's say we got a data matrix $X_{n,p}$ wich represent $n$ person with $p$ feature. We can represent each person in an $R^n$ space, then we get a data cloud.

One goal of PCA is to find the "best approximation" of the data cloud in a $k$ dimensional subspace of $R^n$. This best approximation is introduced in various geometric way:

  1. We look for a projection $P_{E_k}$ such that the sum of the distance between the "projected person" would be maximum.

  2. Similar to "least square"method we look for an "orthogonal projection" which would minimize the sum of distance between the points and their projected.

  3. We look for the maximum "inertie" or Variance of the projected cloud which geometrically give the total distance between the projected cloud and his bartcenter.

  4. Finally a non geometric approach lead us to look for the "best" low rank (rank $k $) approximation of $X $ (cf.

Eckart-Young theorem. My questions are the following (please don't miss the drawing at the end)

A) Could you prove the equivalence between all theses statement?

B) In 1) we consider all the projection whereas in 2) we focused on orthogonal projection. How is that possible ? How do we finally get that the "best" projection is an orthogonal one?

PS: All my question raised from a really interesting presentation wich suffer to prove nothing: http://bertrand.michel.perso.math.cnrs.fr/Enseignements/5MS04/ACP-AFC-ACM.pdf

Geometric motivation

Could you prove the equivalence of the 4 point of view ?

Thanks

$\endgroup$
3
  • $\begingroup$ "Best approximation" is not a very good description as wellness of fit can be defined in very many different ways. Finding the orientations of largest variance is a better description of what PCA and Karhunen-Loeve Transforms do. It is true we get the best fit in a least squares / $L_2$ energy sense, but that is far from the most relevant error measures for many kinds of problems. $\endgroup$
    – mathreadler
    Aug 23, 2017 at 17:58
  • $\begingroup$ I know that is not optimal but i would like first undrrstand it and after consider better way to do. $\endgroup$
    – curious
    Aug 26, 2017 at 6:29
  • $\begingroup$ The link to the PDF is broken. $\endgroup$
    – Rodrigo de Azevedo
    Apr 9, 2021 at 11:06

1 Answer 1

Reset to default
1
$\begingroup$

PCA is "Best" in the sense it preserves most of the energy of the data.
The best way to answer your questions is to analyze the Principal Component Analysis (PCA) using the Singular Value Decomposition (SVD).

Then you'll be able to understand what PCA really does.
Basically, if look at your data as a matrix, the PCA (Using the SVD) is creating the best low rank approximation of the data.

Best in the sense it preserves most f the energy for given rank where the energy is measured by the Singular Values of the data.

You may have a look in my Singular Value Decomposition (SVD) Presentation.
Specifically pages 35-43 and 80-86.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .