Least-squares Projection

1. In least-squares, say we have $n$-points in 2-D space. Now, assume these points don't lie on a line(2-D hyperplane). Do we find $n$-dimensional hyperplane on which all these points lie?

2. If yes, then say I want a line(2-D hyperplane) which is a least-squares approximation for the $n$ points. Is that 2-D hyperplane a projection of the n-dimensional hyperplane we found earlier.

Say, after solving I obtained the b and m. Questions:

1. What the following vector subspace represent? $$m \begin{bmatrix} x_1 \\ x_2 \\ \vdots\\ x_n \end{bmatrix} +b\begin{bmatrix} 1 \\ 1 \\ \vdots \\ y_n\end{bmatrix}$$

2. How the does the line in $R^2$ ,i.e., $y=mx+b$, relate to above columnspace?

• Your question is confusing. When you say $n$-points, do you mean points in $n$-space, or do you mean $n$ points in the plane? If you have points in $n$-space, how many do you have? Far more than $n$? You have to decide what you're trying to model here. You can do all sorts of least-squares fits of data. – Ted Shifrin Dec 4 '16 at 17:23

Assuming you're trying to do linear regression for $n$ points in $\Bbb R^2$, you are seeking a line of the form $y=mx+b$ that is the "best fit." This is often set up as a standard calculus question, minimizing the square-error. As a linear algebra problem, you consider the system of linear equations $mx_i + b = y_i$, $i=1,\dots,n$, and set this up as a matrix problem: $$\begin{bmatrix} x_1 & 1 \\ x_2 & 1 \\ \vdots & \vdots \\ x_n & 1\end{bmatrix}\begin{bmatrix} m \\ b \end{bmatrix} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n\end{bmatrix}, \quad A\mathbf x = \mathbf y, \text{ for short}.$$ Assuming the $n$ data points do not lie on a line, this linear system is inconsistent. We find the least squares solution by projecting the vector $\mathbf y$ onto the column space (image) of $A$. So this is, in essence, doing a projection in $\Bbb R^n$ onto a $2$-dimensional linear subspace.
But there are no $n$-dimensional hyperplanes in this story when your points live in $\Bbb R^2$.
If you try to do quadratic (or higher) regression, then you are looking at $3$-dimensional (and higher) linear subspaces of $\Bbb R^n$ given by the column spaces of matrices with more columns.
• I understood "We find the least squares solution by projecting the vector $\mathbf y$ onto the column space (image) of $A$." But I didn't understand,"So this is, in essence, doing a projection in $\Bbb R^n$ onto a $2$-dimensional linear subspace." What is the $2$-dimensional linear subspace. – Abhishek Bhatia Dec 4 '16 at 18:09
• The column space of $A$ is the $2$-dimensional subspace. :) – Ted Shifrin Dec 4 '16 at 18:11
• Ok, thanks that makes sense. The problem is I can visualize the solution,that is, the line in $R^2$. And for each point, in $R^2$ it is easy to see the projection on the line and error incurred. But in $R^n$, I almost can't visualize what anything means(the projection obtained on $R^n$, the error vector). What does each axis of $R^n$ represent? A data point? – Abhishek Bhatia Dec 4 '16 at 19:26
• You're not projecting on the line in $\Bbb R^2$. You're looking at the vertical displacements from your data points to the line and minimizing the sum of their squares. – Ted Shifrin Dec 4 '16 at 19:35
• I do not have a good answer for what those vectors mean in $\Bbb R^n$. But the projection of $\mathbf y$ onto that subspace is given by specific values $\bar m$ and $\bar b$, and those are the values we use as the coefficients in our equation for the least-squares line. – Ted Shifrin Dec 4 '16 at 19:57