I am looking at the Matrix Cookbook. From my real analysis background, my understanding of calculating derivatives involving matrices is to use the Fréchet derivative on the normed space $(\mathbb{R}^{n \times n}, \|\cdot\|_{op})$ and whatever the target space is, but I am having a hard time linking this to what is used in this book.

For example, consider the matrix trace $\text{Tr}: \mathbb{R}^{n \times n} \rightarrow \mathbb{R}$. This is a linear map so the Frechet derivative in direction ${\bf{V}} \in \mathbb{R}^{n \times n}$ is just the linear map itself independent of the point ${\bf{X}} \in \mathbb{R}^{n \times n}$ so

$$\text{d}\text{Tr}({\bf{X}}){\bf{V}} = \text{Tr}({\bf{V}})$$

whereas in the Matrix Cookbook, the following identity is stated

$$\displaystyle\frac{\partial}{\partial {\bf{X}}}\text{Tr}({\bf{X}}) = {\bf{I}}$$

Which I suppose has the same property of being independent of ${\bf{X}}$ but it's not the same. Another example is the function $f({\bf{X}}) = {\bf{X}}^{-1}$, which has Frechet derivative $\text{d}f({\bf{X}}){\bf{V}} = -{\bf{X}}^{-1}{\bf{V}}{\bf{X}}^{-1}$, MC states an extremely similar looking identity: $$\frac{\partial{\bf{X}}^{-1}}{\partial x} = -{\bf{X}}^{-1}\frac{\partial{\bf{X}}}{\partial x}{\bf{X}}^{-1}$$

My question is, what is the definition of $\displaystyle\frac{\partial}{\partial {\bf{X}}}$, $\partial{\bf{X}}$ and exotic expressions such as $\partial{\lambda_i}$ $\partial{\bf{v}}_i$ (where $\lambda_i, {\bf{v}}_i$ are the eigenvalues and vectors of a real symmetric matrix). I am also curious if there a nice geometric interpretation or analogue to derivatives in Banach spaces and why might these specialised derivatives be preferred over a Frechet derivative in applications.

I have found a similar question with some answers here but I did not find these particularly enlightening, any insightful answers are much appreciated.

  • 2
    $\begingroup$ Consider the trace. Its directional derivative is $\mbox{tr} (V)$, which is the Frobenius inner product $\langle V, I \rangle$. The derivative in the Matrix Cookbook is the $\square$ in the inner product $\langle V, \square \rangle$. $\endgroup$ – Rodrigo de Azevedo Jun 24 '18 at 14:38

Notice the disclaimers on the 2nd page:

"The project of keeping a large repository of relations involving matrices is naturally ongoing"

"Disclaimer: The identities, approximations and relations presented here were obviously not invented but collected, borrowed and copied from a large amount of sources. These sources include similar but shorter notes found on the internet and appendices in books - see the references for a full list"

A collection like that is bound to be sometimes confusing and (self-)contradictory.

1) It seems that when there is an inner product the formulas in MC give you the gradient. That's your first example with trace, see @Rodrigo's comment.

2) When they can't use any inner product to convert it to a gradient, they seem to give you the derivative as if using chain rule and writing $\tfrac{\partial}{\partial X}$ instead of $d$, for example,

$$ d(X^{-1})=-X^{-1}\; dX\; X^{-1} $$ where of course $dX=id$, and so $$ d(X^{-1})(V)=-X^{-1}\; V\; X^{-1} $$

3) And then there are also formulas given in a coordinate-form, with indices etc. or with eigenvalues etc.

I consider it a good raw resource of reference formulas that are not incorrect. I just know that I first need to rewrite or better, re-derive them myself.

| cite | improve this answer | |
  • $\begingroup$ Thank you for the explanation, if possible do you know how to derive $\partial \lambda_i = {{\bf{v}}_i}^T \partial {\bf{A}} {\bf{v}}_i $ and $\partial {\bf{v}}_i = (\lambda_i {\bf{I}}-{\bf{A}})^+ \partial {\bf{A}} {\bf{v}}_i$ ? I can't see how they were obtained as either a partial derivative w.r.t a entry or some gradient. $\endgroup$ – Theo Diamantakis Jun 25 '18 at 14:44
  • $\begingroup$ For the eigenvalue, by differentiating both sides of $v_i^T\lambda_i v_i=v_i^T A v_i$. $\endgroup$ – rych Jun 25 '18 at 14:50
  • 1
    $\begingroup$ More details at math.stackexchange.com/questions/2588473/… $\endgroup$ – rych Jun 26 '18 at 13:39
  • $\begingroup$ How do you re-derive them? Is there a good book for learning this material? I've had linear algebra, and multivariable calculus, but I've never taken derivatives of matrix algebra expressions with respect to matrices. According to the Wikipedia page on 'Matrix Calculus', "The three types of derivatives that have not been considered are those involving vectors-by-matrices, matrices-by-vectors, and matrices-by-matrices. These are not as widely considered and a notation is not widely agreed upon." I need a book that does consider them. Any suggestions? $\endgroup$ – Joe Feb 3 at 17:50
  • $\begingroup$ Often the (Fréchet) derivative is a first chapter in university textbooks on differential geometry, functional analysis, and so on. See also the Reference section on that wiki page. It's a generalization from matrices, vectors, real, complex variables, etc. to the normed vector spaces. $\endgroup$ – rych Feb 4 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.