1
$\begingroup$
  1. In least-squares, say we have $n$-points in 2-D space. Now, assume these points don't lie on a line(2-D hyperplane). Do we find $n$-dimensional hyperplane on which all these points lie?

  2. If yes, then say I want a line(2-D hyperplane) which is a least-squares approximation for the $n$ points. Is that 2-D hyperplane a projection of the n-dimensional hyperplane we found earlier.

EDIT: New Questions after reading the answer.

Say, after solving I obtained the b and m. Questions:

  1. What the following vector subspace represent? $$m \begin{bmatrix} x_1 \\ x_2 \\ \vdots\\ x_n \end{bmatrix} +b\begin{bmatrix} 1 \\ 1 \\ \vdots \\ y_n\end{bmatrix}$$

  2. How the does the line in $R^2$ ,i.e., $y=mx+b$, relate to above columnspace?

$\endgroup$
1
  • $\begingroup$ Your question is confusing. When you say $n$-points, do you mean points in $n$-space, or do you mean $n$ points in the plane? If you have points in $n$-space, how many do you have? Far more than $n$? You have to decide what you're trying to model here. You can do all sorts of least-squares fits of data. $\endgroup$ – Ted Shifrin Dec 4 '16 at 17:23
2
$\begingroup$

Assuming you're trying to do linear regression for $n$ points in $\Bbb R^2$, you are seeking a line of the form $y=mx+b$ that is the "best fit." This is often set up as a standard calculus question, minimizing the square-error. As a linear algebra problem, you consider the system of linear equations $mx_i + b = y_i$, $i=1,\dots,n$, and set this up as a matrix problem: $$\begin{bmatrix} x_1 & 1 \\ x_2 & 1 \\ \vdots & \vdots \\ x_n & 1\end{bmatrix}\begin{bmatrix} m \\ b \end{bmatrix} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n\end{bmatrix}, \quad A\mathbf x = \mathbf y, \text{ for short}.$$ Assuming the $n$ data points do not lie on a line, this linear system is inconsistent. We find the least squares solution by projecting the vector $\mathbf y$ onto the column space (image) of $A$. So this is, in essence, doing a projection in $\Bbb R^n$ onto a $2$-dimensional linear subspace.

But there are no $n$-dimensional hyperplanes in this story when your points live in $\Bbb R^2$.

If you try to do quadratic (or higher) regression, then you are looking at $3$-dimensional (and higher) linear subspaces of $\Bbb R^n$ given by the column spaces of matrices with more columns.

$\endgroup$
6
  • $\begingroup$ I understood "We find the least squares solution by projecting the vector $\mathbf y$ onto the column space (image) of $A$." But I didn't understand,"So this is, in essence, doing a projection in $\Bbb R^n$ onto a $2$-dimensional linear subspace." What is the $2$-dimensional linear subspace. $\endgroup$ – Abhishek Bhatia Dec 4 '16 at 18:09
  • 1
    $\begingroup$ The column space of $A$ is the $2$-dimensional subspace. :) $\endgroup$ – Ted Shifrin Dec 4 '16 at 18:11
  • $\begingroup$ Ok, thanks that makes sense. The problem is I can visualize the solution,that is, the line in $R^2$. And for each point, in $R^2$ it is easy to see the projection on the line and error incurred. But in $R^n$, I almost can't visualize what anything means(the projection obtained on $R^n$, the error vector). What does each axis of $R^n$ represent? A data point? $\endgroup$ – Abhishek Bhatia Dec 4 '16 at 19:26
  • 1
    $\begingroup$ You're not projecting on the line in $\Bbb R^2$. You're looking at the vertical displacements from your data points to the line and minimizing the sum of their squares. $\endgroup$ – Ted Shifrin Dec 4 '16 at 19:35
  • 1
    $\begingroup$ I do not have a good answer for what those vectors mean in $\Bbb R^n$. But the projection of $\mathbf y$ onto that subspace is given by specific values $\bar m$ and $\bar b$, and those are the values we use as the coefficients in our equation for the least-squares line. $\endgroup$ – Ted Shifrin Dec 4 '16 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.