In a book i'm reading there are these formulas, $y$ and $\beta$ are vectors of same dimensions, $X$ is a matrix:

$$RSS(\beta) = (y - X\beta)^T(y - X\beta)$$ $$\frac{\partial RSS(\beta)}{\partial \beta} = -2X^T(y - X\beta)$$ $$\frac{\partial^2 RSS(\beta)}{\partial \beta \partial \beta^T} = 2X^TX$$

From what I understand, when he uses $\partial \beta$ as denominator, it uses the numerator layout convention, meaning the derivative is actually calculated w.r.t $\partial \beta^T$, and when $\partial \beta^T$ is used explicitly, it is just the same as using $\partial \beta$ with the numerator layout.

If that is correct, then why is he using two layouts at the same time in the second order derivative, why not just write $\partial \beta^2$, or even $\partial \beta^T\partial \beta$ ? If using $\partial \beta$ or $\partial \beta^T$ as denominator produces the same result, why not always be explicit and use $\partial \beta^T$ to avoid confusion ? Is there any advantage to have multiple layouts for matrix derivatives ?

  • $\begingroup$ Note, these equations appear on page 45 of The Elements of Statistical Learning by Hastie, Tibshirani and Friedman, 2e. Naming the text may help others find this question. $\endgroup$ – mb7744 Jun 26 '18 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.