0
$\begingroup$

I was trying to follow along with the derivation of the normal equations, and almost everything here makes sense. However, when it came to the differentiation, I got stuck at how one goes from $$S(\beta) = y^Ty-2\beta^TX^Ty + \beta^TX^TX\beta$$

to

$$-X^Ty+(X^TX)\beta=0$$

just by computing $\frac{\partial S(\beta)}{\partial \beta} = 0$. I'm not very familiar with matrix calculus but I do want to understand how each step in this derivation works and this is the main part I'm getting stuck on. I speculate that one can cancel $\beta^T$ from both terms because it's equated to 0, but I can't see how the $2$ drops off.

$\endgroup$
0
$\begingroup$

We have: $$S(\beta+\delta \beta)-S(\beta)= -2 (\delta \beta)^T X^T y +(\delta \beta)^T X^T X \beta +\beta^T X^T X \delta \beta + (\delta \beta)^T X^T X \delta \beta$$ as $\beta^T X^T X \delta \beta$ is a $1 \times 1$ matrix we have: $$\beta^T X^T X \delta \beta = \left( \beta^T X^T X \delta \beta \right)^T$$

which give: $$S(\beta+\delta \beta)-S(\beta)= (\delta \beta)^T \left(-2X^T y +2 X^T X \beta \right)+O(\Vert \delta \beta \Vert^2)$$

$\endgroup$
0
$\begingroup$

Take a look here https://en.wikipedia.org/wiki/Matrix_calculus for details on matrix calculus. In a nutshell, $ \beta ' X'X \beta$ is a quadratic form with a symmetric matrix $X'X$, hence $$ \frac{\partial \beta ' X'X \beta}{\partial \beta} = 2 X'X \beta, $$ and $\beta ' X'y$ is of the form $\beta ' a$, where $a$ is $p \times 1$ vector, hence $$ \frac{\partial \beta ' X'y}{\partial \beta} = X'y. $$

To prove these formulas just expand explicitly the expressions and take partial derivatives w.r.t. to each $\beta_j$. After that, return to a matrix representation and you'll get the result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.