Cross posted from stats.stackexchange as it wasn't getting any love (given it a few days)!

In the context of REML estimation there is the result (ignoring some constants) that (my interest is in the matrix algebra so some notation is suppressed):

$l(\mathbf V_0)=\log |\mathbf V_0| + \text{tr}(\mathbf V_0^{-1}\mathbf S) \tag{1}$

Where both $\mathbf V_0$ and $\mathbf S$ are symmetric and invertible. I am told that the following expression can be obtained from (1) by differentiating with respect to a parameter $\sigma_i$ of $\mathbf V_0$:

$\text{tr}\left[\mathbf V_0 \frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}\right]-\text{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right] \tag{2}$

I'm trying to see what steps got this, but am stuck.

Now I'm obviously lacking some machinery for dealing with these things but I don't know where to look. The issue is that, say, both terms in (1) are scalar, and we are differentiating w.r.t a scalar, so the it would seem the answer should be scalar. But my workings end up matrix valued. E.g.

$\frac{\partial \log |\mathbf V_0|}{\partial \sigma_i} = \mathbf V_0^{-1} \frac{\partial \mathbf V_0}{\partial \sigma_i} $

Clearly the trace gets involved wrapping it all up into a scalar, but I don't know how or why!

  • $\begingroup$ I believe this is because the chain rule in higher dimensions generalizes to the use of an inner product (or trace, in this case). See en.wikipedia.org/wiki/…. $\endgroup$ – Ross B. Jun 3 '14 at 17:41

Assuming that $\mathbf S$ is a constant matrix, your formula is wrong. For a counterexample, consider the scalar case $\mathbf V_0=\sigma_i=\sigma$ and $S=1$. We have $l(\mathbf V_0)=\log\sigma+\frac1\sigma$ and $$ \frac{d\,l(\mathbf V_0)}{d\sigma}=\frac1\sigma-\frac1{\sigma^2} =-\left[-\frac1{\sigma^2}(\sigma-1)\right] =-\operatorname{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right]. $$ In other words, your formula has the first summand in excess.

The partial derivative of $l(\mathbf V_0)$ with respect to $\sigma_i$ should be $-\operatorname{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right]$. To prove this, recall that the derivative of determinant and derivative of matrix inverse of a matrix function $A$ of a scalar parameter $x$ are given by \begin{align} \frac{d|A|}{dx} &= |A| \operatorname{tr}\left(A^{-1} \frac{dA}{dx}\right),\tag{1}\\ \frac{dA^{-1}}{dx} &= -A^{-1}\frac{dA}{dx}A^{-1}.\tag{2} \end{align} Therefore $$ \frac{d\log|A|}{dx} =\frac1{|A|}\frac{d|A|}{dx} =\operatorname{tr}\left(A^{-1} \frac{dA}{dx}\right) =-\operatorname{tr}\left(\frac{dA^{-1}}{dx}A\right). $$

  • $\begingroup$ Thanks, that helps. One question: any suggestions where can I look up more about the use of the trace at the end? How is it that certain matrix derivatives end up as a trace? Much appreciated. - Actually hold that, following some links in the wikipedia page you posted gets me somewhere promising! $\endgroup$ – conjectures Jun 4 '14 at 20:22
  • $\begingroup$ @conjectures I haven't any suggestions, sorry. By the way, what ends up being the trace is not a matrix derivative, but the derivative of a matrix determinant. That the derivative of a scalar function is a scalar function is not surprising. $\endgroup$ – user1551 Jun 5 '14 at 11:01

Rewrite the function into a form more amenable to differentiation $$\eqalign{ f &= \log(\det(V)) + {\rm tr}(V^{-1}S) \cr &= {\rm tr}(\log(V)) + S:V^{-1} \cr }$$ where the colon (:) denotes the Frobenius product.

Letting $d$ denote the derivative operator with respect to $\sigma_i\,\,$ differentiate the above to obtain $$\eqalign{ df &= V^{-1}:dV - S:V^{-1}\,dV\,V^{-1} \cr &= V^{-1}:dV - V^{-1}SV^{-1}:dV \cr &= (V^{-1} - V^{-1}SV^{-1}):dV \cr &= V^{-1}(V-S)V^{-1}:dV \cr &= {\rm tr}\big[V^{-1}(V-S)V^{-1}\,dV\big] \cr }$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.