# Is the least squares solution to an overdetermined system a triangle center?

My question relates to a simple case of the least squares problem. For example, take the system of equations \begin{align*} 2x- y &= 2, \\ x + 2y &= 1, \\ x+y &= 4, \end{align*} which is clearly overdetermined. Using a least squares approach, we can solve the system $$A^TA\mathbf{x^*} = A^T\mathbf{b},$$ where $\mathbf{x^*}$ minimises $\|\ A\mathbf{x} -\mathbf{b} \ \|$, and $$A = \begin{bmatrix} 2 & -1\\ 1 & 2 \\ 1 & 1 \end{bmatrix} \ , \quad \mathbf{b}= \begin{bmatrix} 2 \\ 1 \\ 4 \end{bmatrix} \quad \mbox{and} \quad \mathbf{x}= \begin{bmatrix} x \\ y \end{bmatrix},$$ to obtain an approximation for the solution as $\left(\frac{10}{7}, \frac{3}{7} \right)$.

If you plot the three straight lines that constitute the system of equations, they intersect at three points that form the vertices of a triangle.

My question is - does the solution always fall within this triangle (as in this example), and if so, is this point a (non-traditional) triangle center?

• I believe the result will always be the barycenter of the triangle, but do not have a proof. – Alex Becker Aug 20 '13 at 6:47
• The trouble is that scaling one of the equations does not change the triangle but it does change the least-squares solution. This is the difference between the geometry and the algebra of the problem. If you scale everything so that the rows of $A$ are unit vectors, you may get a meaningful triangle center. – Rahul Aug 20 '13 at 6:56
• It's not the barycentre. What do you get when two of the lines are nearly parallel? – Robert Israel Aug 20 '13 at 7:02
• Good question. Suppose we "normalize" all the line equations -- we write them in the form $ax+by=c$, where $a^2+b^2=1$. Then I think you're asking ... what point in the interior of a triangle minimises the sum of the (squared?) distances to its three sides. I don't know the answer, but I'd like to know. – bubba Aug 20 '13 at 7:36
• The point minimizing the sum of the squares of the distances to the sidelines of a triangle is the symmedian point of the triangle. See en.wikipedia.org/wiki/Symmedian for the definition, and Exercise 7.3 in Ross Honsberger's "Episodes in Nineteenth and Twentieth Century Euclidean Geometry" (but ignore the word "inside", which isn't really needed) for the proof of the above property. Basically it follows from the Cauchy-Schwarz inequality, along with the fact that the distances from the symmedian point to the sidelines are proportional to the length of these sides. – darij grinberg Aug 20 '13 at 8:32

As Rahul Narain commented, the precise position of the least-squares solution depends on the scaling of the individual equations: as you scale up an equation, so does the importance of the distance of your solution to the corresponding line, and the least-squares solution will end up closer to that line. You can say that in any case the least-squares solution will lie in the interior of the triangle formed by the three lines, since for a point in any of the other $6$ sectors into which the lines cut up the plane, moving in some direction will decrease the distances to all three lines at the same time, so one can never have a minimum there.
In the spacial case where you scale the equations in such a way that for each of the the sum-of-squares-of-coefficients gives the same value (for instance$~1$), then you can describe the least-square solution as the unique point$~P$ such that the sum of the three vectors from $P$ to its orthogonal projections on the three lines is zero; in other words $P$ is the barycenter of its own pedal triangle (the triangle formed by those projections). That this is so can easily be seen by taking the formula implicit in the comment by @bubba and differentiating it.