Can someone explain to me least squares regression using vectors?

Question

I am reading Foundations and Applications of Stats by Pruim. I am having a tough time understanding least squares regression from a vector notation standpoint. I don't have a good understanding of linear algebra.

I have a few questions:

1. Why does the residual vector (red line) have to point to and end at the same point as the observation vector (black line)? I get that the red line has to be orthogonal to the model space as we look to minimize the length of the residual and so theoretically, the red line has to be the shortest distance between the model space and the black line. But if we subtract two vectors, [y - yhat], I don't get how the length and magnitude of the resulting vector has to be that specific red line?
2. What does it mean by [y - y^] must be shorter than all other [y - y~]. I mean, if [y~ = y], wouldn't that lead to a vector length of 0? What other constraint or assumption am I missing?

Answer 1

1. Saying $\mathbf e = \mathbf y-\mathbf{\hat y}$, is equivalent to saying $\mathbf y = \mathbf{\hat y} + \mathbf e$. This literally indicates that $\mathbf e$ starts at $\mathbf{\hat y}$ and ends at $\mathbf y$.
2. The whole point of the diagram is that the fit vector, $\mathbf{\hat y}$ must come from the model space. It's possible for the observation vector $\mathbf y$ to also be in the model space, but most of the time it is, as shown in your diagram, not. Yes, $\mathbf{\hat y} = \mathbf y$ would lead to a residual vector of length $0$. But, when $\mathbf y$ is not in the model space (which is usually the case), then $\mathbf{\hat y} = \mathbf y$ is not an option.