2
$\begingroup$

I find a derivation of the least square estimator for multiple linear regression, but there some part I am not fully understand some part in the. The derivation is following:

Starting from $y= Xb +\epsilon $, which really is just the same as

$\begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{N} \end{bmatrix} = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1K} \\ x_{21} & x_{22} & \cdots & x_{2K} \\ \vdots & \ddots & \ddots & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{NK} \end{bmatrix} * \begin{bmatrix} b_{1} \\ b_{2} \\ \vdots \\ b_{K} \end{bmatrix} + \begin{bmatrix} \epsilon_{1} \\ \epsilon_{2} \\ \vdots \\ \epsilon_{N} \end{bmatrix} $

let $e = y - Xb$, then it all comes down to minimzing $e'e$:

$e'e = \begin{bmatrix} e_{1} & e_{2} & \cdots & e_{N} \\ \end{bmatrix} \begin{bmatrix} e_{1} \\ e_{2} \\ \vdots \\ e_{N} \end{bmatrix} = \sum_{i=1}^{N}e_{i}^{2} $

So minimizing $e'e$ gives us:

$min_{b}$ $e'e = (y-Xb)'(y-Xb)$

$min_{b}$ $e'e = y'y - 2b'X'y + b'X'Xb$

$\frac{\partial(e'e)}{\partial b} = -2X'y + 2X'Xb \stackrel{!}{=} 0$

$X'Xb=X'y$

$b=(X'X)^{-1}X'y$

my problem is about this part "$min_{b}$ $e'e = (y-Xb)'(y-Xb)$", I believe the ' notation at here is the transpose, but I could not get the result same as above, if I just open the bracket as usual \begin{align*} min_{b}e'e &= (y-Xb)'(y-Xb)\\ & = y'y - y'Xb - b'X'y + b'X'Xb \end{align*}which is not same as what the derivation got "$min_{b}$ $e'e = y'y - 2b'X'y + b'X'Xb$", could anyone explain to me how could the derivation get this matrix equation?

$\endgroup$
1
$\begingroup$

This is because $$b'X'y = (y'Xb)'$$ combined with the fact that they are both scalar. A transposed scalar equals itself.

$\endgroup$
1
  • $\begingroup$ OH yeah, thanks, I didn't aware of that. $\endgroup$ – Snow Max Feb 23 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.