0
$\begingroup$

I have read about two different approaches to calculating a least-squares fit to points in 2D, and I'm having trouble seeing how these two approaches are related.

So, suppose we have $n$ 2D points $(x_1,y_1), \ldots, (x_n, y_n)$ that we want to fit with a straight line. I want a so-called total least squares fit, i.e. I want to minimize the sums of squares of orthogonal distances to the line.

Let's suppose the data points have already been normalized so that $\sum x_i = 0$ and $\sum y_i = 0$. Then it's pretty easy to show that the best fit line passes through the point $(0,0)$ and so it has the form $Ax + By = 0$, and we just need to find $A$ and $B$.

This paper gives one way to calculate $A$ and $B$. We define $$ a = \sum x_i^2 - \sum y_i^2 \quad\quad ; \quad\quad b = \sum x_iy_i $$ and then $$ A = 2b \quad\quad ; \quad\quad B = - a - \sqrt{a^2 + 4b^2} $$ OK. Cool. Nice and simple.

But a more common approach (like here, for example) is to form the matrix $$ M = \begin{bmatrix} x_1 & \ldots & x_n \\ y_1 & \ldots & y_n \\ \end{bmatrix} $$ and then the normal vector $(A,B)$ of the best fit line is given by an eigenvector of $MM^T$ corresponding to its smallest eigenvalue.

So, my question: the simple formulas for $A$ and $B$ are somehow magically telling me about the eigenvectors of a matrix. How can that be? I don't see the connection at all.

$\endgroup$

1 Answer 1

2
$\begingroup$

For the case of a $2 \times 2$ matrix $M$, we have the two facts

$$\lambda_1+\lambda_2= trace(M)$$ and $$\lambda_1\lambda_2= det(M)$$

where $\lambda_1,\lambda_2$ are the two eigenvalues. That's the "magic connection".

There's still work you would have to do to compute the trace and the det. And then you would have to solve for the two eigenvalues from knowing their sum and their product (Hint: eliminate one of them, and you should end up with a quadratic equation in the other. Your formula for $B$ sure looks like an application of the quadratic formula.)

It's a bit messy, but we are able to get those fairly simple formulas in the end. It's only because we're dealing with a $2\times 2$ matrix, where we have fairly nice equations obeyed by the eigenvalues - it would all be considerably more messy for even the $3\times 3$ case.

[Note : I confess that I didn't carry out the math just now to verify that it works exactly how I've described. But I'm pretty sure I did this before, and I'd bet a few bucks that it does.]

$\endgroup$
7
  • $\begingroup$ Thanks. That gives me some insight into the eigenvalues, but less for the eigenvectors. But, very helpful. $\endgroup$
    – bubba
    Jan 23, 2023 at 3:30
  • $\begingroup$ Hmm, I did read down fairly quickly and thought "I can provide some insight on the question in bold", and kind of skipped over the vector part. I wonder if the key point about the eigenvalues ("you can find explicit formulas for the 2x2 case") might not also be true for the vectors. I already knew the facts about the eigenvalues, but maybe a quick search could turn up explicit formulas for the vectors in the 2x2 case? $\endgroup$
    – JonathanZ
    Jan 23, 2023 at 3:38
  • $\begingroup$ My fault. I meant to write "eigenvectors" in the bold final question, not "eigenvalues". I edited. $\endgroup$
    – bubba
    Jan 23, 2023 at 3:47
  • 1
    $\begingroup$ I just noticed that Mathematica's "Eigenvectors" command gives explicit formulas for the case of 2x2 matrices. I swear I tried this before. $\endgroup$
    – bubba
    Jan 23, 2023 at 3:48
  • 1
    $\begingroup$ And this ... people.math.harvard.edu/~knill/teaching/math21b2004/exhibits/… $\endgroup$
    – bubba
    Jan 23, 2023 at 3:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .