Matrix formulation is straightforward:

$\mathbf{y} = \mathbf{X} \boldsymbol{\hat{\beta}} +\boldsymbol{\hat{\varepsilon}}$

cost function: $E = {\boldsymbol{\hat{\varepsilon}}}^T{\boldsymbol{\hat{\varepsilon}}} = {(\mathbf{y} - \mathbf{X}\boldsymbol{\hat{\beta}})}^T(\mathbf{y} - \mathbf{X}\boldsymbol{\hat{\beta}})$

... differentiating wrt $\boldsymbol{\hat{\beta}}$ and searching for extremum:

$\frac{\partial E}{\partial \boldsymbol{\hat{\beta}}} = 2 \mathbf{X}^T\mathbf{X} \boldsymbol{\hat{\beta}} - 2 \mathbf{X}^T \mathbf{y} = 0$

thus the OLS estimate of $\boldsymbol{\hat{\beta}}$ is: $\boldsymbol{\hat{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$

So, there is probably some limitation to the previous relation (e.g. $(\mathbf{X}^T\mathbf{X})^{-1}$ have to exist) ... am I right?

If I try to make the same with component notation, there is a problem in the same formula (I'll come back later to this).

In component formalism (using Einstein's summation convention):

$E = (X_{ij} \beta_j - y_i)^2 = (X_{ij}\beta_j)^2 - 2 X_{ij}\beta_j y_i + y_i^2$

$\frac{\partial E}{\partial \beta_j} = 2X_{ij} \beta_j X_{ij} - 2X_{ij} y_i = 0$

$X_{ij} \beta_j X_{ij} = X_{ij} y_i$

Now, every term is just scalar, so it's tempting to cancel $X_{ij}$ on both sides. However, this just leads to trivial relation: $y_i = X_{ij} \beta_j$

Can someone help me to enlighten this, please? Isn't it somehow connected to the use of only lower indices? When I have to consider both lower and upper indices (tensors and duals)?

Thank you!


The error in your component derivation: When you differentiate wrt $\beta_j$, the index $j$ now has two roles: one as the generic index in the sum, and one as the index specifying which $\beta$ you are differentiating with respect to. The partial derivative wrt $\beta_j$ should treat the other $\beta$'s as constant, but your notation can no longer distinguish them!

Better to use a new index, say $k$, to perform the differentiation. When you do this, you find that the partial derivative will be $$\frac{\partial E}{\partial\beta_k}=2X_{ij}\beta_jX_{ik} - 2X_{ik}y_i.\tag1$$ There's still summing going on (involving $i$ and $j$, with $k$ held constant) when you set (1) to zero, so it doesn't make sense to factor out $X_{ij}$. Convert back into matrix notation and you'll get $$X^TX\beta=X^Ty.\tag2$$ Specifically, $\sum_i\sum_jX_{ij}\beta_jX_{ik}$ is the $k$th member of the vector $X^TX\beta$, while $\sum_iX_{ik}y_i$ is the $k$th member of $X^Ty$.

  • $\begingroup$ Thank you! That was really helpful. $\endgroup$ – user847643 Nov 12 '20 at 10:08

What you have is essentially $\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}^T\boldsymbol{y}$, regardless of which notation you use. As you pointed out, if you cancel $\boldsymbol{X}^T$ from both sides, you are left with $\boldsymbol{X}\boldsymbol{\beta}=\boldsymbol{y}$. However, the point of doing least squares in the first place is that $\boldsymbol{X}\boldsymbol{\beta}=\boldsymbol{y}$ cannot be solved for $\boldsymbol{\beta}$, i.e., $\boldsymbol{X}$ is not invertible. By multiplying each side by $\boldsymbol{X^T}$, you get $\boldsymbol{X}^T\boldsymbol{X}$ on the left, which is invertible.

  • $\begingroup$ Thank you! Hmm, thus it would be possible to look at OLS from an algebraic perspective (as you said: non-invertible $\bf{X}$ --> fix it with left multiplication) and completely forget the idea of searching for extrema? $\endgroup$ – user847643 Nov 12 '20 at 10:13
  • $\begingroup$ Yes, though you get the same thing as with using calculus. $\endgroup$ – nosuchthingasmagic Nov 14 '20 at 22:18
  • $\begingroup$ To justify the normal equations using linear algebra/geometry more fully, you can start from the fact that $\boldsymbol{X}\boldsymbol{\beta}$ is in the column space of $\boldsymbol{X}$, but $\boldsymbol{y}$ is not in general in the column space, so you find the closest vector to $\boldsymbol{y}$ that is in the column space (i.e., project $\boldsymbol{y}$ into the column space). The error: $\boldsymbol{X}\hat{\boldsymbol{\beta}}-\boldsymbol{y}$ must be perpendicular to the column space, so $\boldsymbol{X}^T(\boldsymbol{X}\hat{\boldsymbol{\beta}}-\boldsymbol{y})=0$. $\endgroup$ – nosuchthingasmagic Nov 14 '20 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.