I use The Matrix Cookbook by Kaare Brandt Petersen and Michael Syskind Pedersen to solve many problems(mostly matrix derivatives). In a most cases, I just map the problem to one of the formula and solve it but I cannot derive the formula by myself easily(I may prove the given formula is correct). Since I do not have access to the book when I am taking test, I am wondering how others perform these kinds of calculation without a reference book. Is this book just considered as a reference or I should study the book and try to drive the formula by myself? Does anyone have an insight on how to get better in matrix calculus(specially derivative with respect to vector or matrix) or how should I study such books? Thanks in advance.

  • $\begingroup$ Could you be specific about a type of calculation you are referring to? Perhaps giving an example problem? $\endgroup$ – TomGrubb Aug 2 '17 at 1:45
  • $\begingroup$ @ThomasGrubb Thanks for your comment. As an example last time for solving the following problem I used formula 119 in the book. Without that I had no idea how to solve it quickly. $E_D(\widetilde{ W}) = \frac{1}{2} Tr\{(\widetilde{X}\widetilde{W}- T)^T(\widetilde{X}\widetilde{W}- T)\}\quad\quad$ Setting the derivative with respect to $\widetilde{W}$ to zero $\widetilde{W} = (\widetilde{X}^T\widetilde{X})^{-1}\widetilde{X}^TT$. $\endgroup$ – Crimson Aug 2 '17 at 1:49
  • $\begingroup$ This example is from Pattern Recognition and Machine Learning book chapter 4.1, page 185, I was trying to do the calculation by myself. $\endgroup$ – Crimson Aug 2 '17 at 1:59
  • $\begingroup$ The best way to memorize a formula is understand its proof. $\endgroup$ – Surb Aug 5 '17 at 12:14
  • $\begingroup$ Thanks for your insight, Surb. $\endgroup$ – Crimson Aug 7 '17 at 1:49

The Matrix Cookbook is only a list of formulas without proof. It absolutely does not learn to understand the theory - and the practice - of derivation. Better, you can have a look at the solutions given in this website. Roll up your sleeves, my friend.

Let $g(U)=1/2tr(U^TU),f(W)=1/2tr((XW-T)^T(XW-T))$.

If $K$ has same dimensions as $U$, then $Dg_U(K)=1/2(tr(K^TU)+tr(U^TK))=tr(K^TU)$.

We deduce that, if $H$ has same dimensions as $W$, then $Df_W(H)=tr((XH)^T(XW-T))=tr(H^TX^T(XW-T))$.

If, for every $H$, $Df_W(H)=0$, then $X^T(XW-T)=0$, that implies

$W=(X^TX)^{-1}X^TT$ (if $X^TX$ is invertible).

  • $\begingroup$ Absolutely agree. The principle (the product of two first-order terms is negligible for first-order approximation) is so simple that I always wonder why anyone would need that "cookbook" for calculating derivatives. $\endgroup$ – user1551 Aug 5 '17 at 14:47
  • 4
    $\begingroup$ @user1551 Am afraid I would have to disagree with your statement, calculating matrix derivatives is an involving one for those who are not well versed in mathematics, having something like a cookbook saves occasional users time. Your statement is remotely analogous to saying why should there be more accessible programming languages like Python when C is so simple. $\endgroup$ – chutsu Mar 25 '18 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.