1
$\begingroup$

I am reading Foundations and Applications of Stats by Pruim. I am having a tough time understanding least squares regression from a vector notation standpoint. I don't have a good understanding of linear algebra.

enter image description here enter image description here enter image description here

I have a few questions:

  1. Why does the residual vector (red line) have to point to and end at the same point as the observation vector (black line)? I get that the red line has to be orthogonal to the model space as we look to minimize the length of the residual and so theoretically, the red line has to be the shortest distance between the model space and the black line. But if we subtract two vectors, [y - yhat], I don't get how the length and magnitude of the resulting vector has to be that specific red line?
  2. What does it mean by [y - y^] must be shorter than all other [y - y~]. I mean, if [y~ = y], wouldn't that lead to a vector length of 0? What other constraint or assumption am I missing?
$\endgroup$

1 Answer 1

1
$\begingroup$
  1. Saying $\mathbf e = \mathbf y-\mathbf{\hat y}$, is equivalent to saying $\mathbf y = \mathbf{\hat y} + \mathbf e$. This literally indicates that $\mathbf e$ starts at $\mathbf{\hat y}$ and ends at $\mathbf y$.
  2. The whole point of the diagram is that the fit vector, $\mathbf{\hat y}$ must come from the model space. It's possible for the observation vector $\mathbf y$ to also be in the model space, but most of the time it is, as shown in your diagram, not. Yes, $\mathbf{\hat y} = \mathbf y$ would lead to a residual vector of length $0$. But, when $\mathbf y$ is not in the model space (which is usually the case), then $\mathbf{\hat y} = \mathbf y$ is not an option.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.