# Confused regarding the interpretation of A in the least squares formula A^T A = A^T b

So I'm was watching gilbert strang's lecture to refresh my memory on least squares, and there's something that's confusing me (timestamp included).

In the 2D case he has $A=[1,1;1,2;1,3]$. In the lecture he talks about how A is a subspace in our n dimesional (in this case n=2) space.

If you look at the top left point of the chalk board in that time stamp you will see he has written down $A=[a_{1},a_{2}]$. Here he was talking about the 3d case, and the $a$s were vectors that spanned a plane onto which we wanted to project.

My problem is, I'm not quite sure how I'm supposed to understand the values for the 2D. Clearly the 2nd column is the x values, but can one say they span a space? The first one is obviously just the constant for our linear equation but how could one interpret that in the context of A spanning the subspace of our 2D world?

Basically I find there's a contradiction between how he views 2D and how he looks at higher dimensions. I don't see how it makes sense for A to be made out of two vectors as columns in 3D and for A to be made out of 2 different columns in 2D.

## 2 Answers

Ignore the fact that the question arose from linear regression. Just think about the space spanned by the vectors $[1,1,1]^T$ and $[1,2,3]^T$. This is a 2D plane living inside 3D space. Solving the least squares form of $Ax=b$ amounts to finding the vector on this plane which is closest to $b$ and writing it as a linear combination of $[1,1,1]^T$ and $[1,2,3]^T$.

• So am I understanding you correctly that even though the problem started out in 2D we ignore that and instead pretty much just take the number of our samples n into n space and look at things from there? Nov 21, 2015 at 22:04
• @Nimitz14 Yes: the geometric problem is two dimensional, but the corresponding algebraic problem is three dimensional, because we have conditions at three points.
– Ian
Nov 21, 2015 at 22:16
• @Nimitz14 That said, once you've understood that much, there is a connection: the square of the distance between the point on the plane and the desired point is the sum of the squares of the vertical residuals in the original picture. This is important: if somehow we had two completely different ways of measuring distances in the two contexts, then least squares would be a bad way of finding lines of best fit.
– Ian
Nov 21, 2015 at 22:22

You have an $n\times 2$ matrix: $$A = \begin{bmatrix} 1 & a_1 \\ 1 & a_2 \\ 1 & a_3 \\ \vdots & \vdots \\ 1 & a_n \end{bmatrix}$$ The two columns span a $2$-dimensional subspace of an $n$-dimensional space. Your scatterplot is $\{(a_i,b_i) : i = 1,\ldots, n\}$; it is a set of $n$ points in a $2$-dimensional space. The least-squares estimates $\hat x_1$ and $\hat x_2$ are those that minimize the sum of squares of residuals: $$\sum_{i=1}^n (\hat x_1 + \hat x_2 a_i - b_i)^2.$$ The vector of "fitted values" has entries $\hat b_i = \hat x_1 + \hat x_2 a_i$ for $i=1,\ldots,n$.

The vector $\hat{\vec b}$ of fitted values is the orthogonal projection of $\vec b$ onto the column space of the matrix $A$.

• This is an answer you should give if you're a student answering a professor. Surprise surprise, I'm not a professor. You're just giving me a bunch of statements with zero deduction in between, how am I supposed to take anything out of this? Nov 21, 2015 at 22:08
• @Nimitz14 : I disagree with your first sentence in your comment. I think if you read it you may find it answers your question. Is there something in particular in it that you can't understand? If so, say so and maybe I can explain it. Nov 21, 2015 at 22:11
• @Nimitz14 : You assume a lot and you are wrong. ${}\qquad{}$ Nov 22, 2015 at 0:00