Finding residual error of fitting a 3D Line I'm trying to think of a way where I can find the residual error after a fitting a 3D line to a point cloud in an efficient way.
If I use the eigenvalue method, is the residual error already calculated? Or should I simply find the perpendicular distance from each point to the line and sum it up (the simple, naive answer)?
 A: Let $X = [x_1,\ldots, x_n]$ be our $d\times n$ data matrix ($n$ points in $\mathbb{R}^d$) with mean 0 (for convenience). 
The direction of the best fit line is given by the  eigenvector of $X^TX$ corresponding to the largest eigenvalue (equivalently, since $X^TX$ is positive-semidefinite, the largest singular value).
Given a point $x_j$, the nearest point on the line spanned by $u$ is simply the projection $(u^Tx) u = uu^Tx$. Therefore, the $j$-th squared residual is
$$
r_j = \Vert x_j - uu^Tx_j \Vert_2^2
= \Vert (I-uu^T)x_j \Vert_2^2
=  x_j^T(I-uu^T)x_j
$$
We want to compute the sum of square residuals,
$$
R = \sum_{j=1}^{d} r_j^2
= \sum_{j=1}^{d} x_j^T(I-uu^T)x_j
$$
I believe that this is equal to (I'll try to post a proof later),
$$
\operatorname{tr}(X^TX - uu^TX^TX)
$$
Therefore,
$$
R = \operatorname{tr}(X^TX - uu^TX^TX) = \sigma_2 + \cdots + \sigma_n
$$
where we have used the fact that since $u$ is the first singular vector of $X^TX$ then $uu^TX^TX$ we know that the eigenvalues/singular values of $X^TX-uu^TX^TX$ are $\sigma_2, \ldots, \sigma_d,0$ (where $\sigma_i = \sigma_i(X^TX)$ is the $i$-th largest singular value of $X^TX$).
Therefore, if you compute the SVD of $X^TX$, the first singular vector gives you the direction of the best fit line, and the sum of the rest of the singular values gives the sum of squared residuals.
This can be easily generalized to the best fit $k$-dimensional space by replacing $u$ with $U = [u_1, \ldots, u_k]$ (in which case the sum of squared residuals is $\sigma_{k+1} + \cdots + \sigma_d$).
Starting at page 10 of this document they state this as a theorem without proof.
