Matrix calculus - derivative of scalar function of matrix function of scalar Cross posted from stats.stackexchange as it wasn't getting any love (given it a few days)! 
In the context of REML estimation there is the result (ignoring some constants) that (my interest is in the matrix algebra so some notation is suppressed):
$l(\mathbf V_0)=\log |\mathbf V_0| + \text{tr}(\mathbf V_0^{-1}\mathbf S) \tag{1}$
Where both $\mathbf V_0$ and $\mathbf S$ are symmetric and invertible. I am told that the following expression can be obtained from (1) by differentiating with respect to a parameter $\sigma_i$ of $\mathbf V_0$:
$\text{tr}\left[\mathbf V_0 \frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}\right]-\text{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right] \tag{2}$
I'm trying to see what steps got this, but am stuck.
Now I'm obviously lacking some machinery for dealing with these things but I don't know where to look. The issue is that, say, both terms in (1) are scalar, and we are differentiating w.r.t a scalar, so the it would seem the answer should be scalar. But my workings end up matrix valued. E.g.
$\frac{\partial \log |\mathbf V_0|}{\partial \sigma_i} = \mathbf V_0^{-1} \frac{\partial \mathbf V_0}{\partial \sigma_i} $
Clearly the trace gets involved wrapping it all up into a scalar, but I don't know how or why!
 A: Rewrite the function into a form more amenable to differentiation
$$\eqalign{
 f &= \log(\det(V)) + {\rm tr}(V^{-1}S) \cr
   &= {\rm tr}(\log(V)) + S:V^{-1} \cr
}$$
where the colon (:) denotes the Frobenius product.
Letting $d$ denote the derivative operator with respect to $\sigma_i\,\,$ differentiate the above to obtain 
$$\eqalign{
df &= V^{-1}:dV - S:V^{-1}\,dV\,V^{-1} \cr
   &= V^{-1}:dV - V^{-1}SV^{-1}:dV \cr
   &= (V^{-1} - V^{-1}SV^{-1}):dV \cr
   &= V^{-1}(V-S)V^{-1}:dV \cr
   &= {\rm tr}\big[V^{-1}(V-S)V^{-1}\,dV\big] \cr
}$$
A: Assuming that $\mathbf S$ is a constant matrix, your formula is wrong. For a counterexample, consider the scalar case $\mathbf V_0=\sigma_i=\sigma$ and $S=1$. We have $l(\mathbf V_0)=\log\sigma+\frac1\sigma$ and
$$
\frac{d\,l(\mathbf V_0)}{d\sigma}=\frac1\sigma-\frac1{\sigma^2}
=-\left[-\frac1{\sigma^2}(\sigma-1)\right]
=-\operatorname{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right].
$$
In other words, your formula has the first summand in excess.
The partial derivative of $l(\mathbf V_0)$ with respect to $\sigma_i$ should be $-\operatorname{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right]$. To prove this, recall that the derivative of determinant and derivative of matrix inverse of a matrix function $A$ of a scalar parameter $x$ are given by
\begin{align}
\frac{d|A|}{dx} &= |A| \operatorname{tr}\left(A^{-1} \frac{dA}{dx}\right),\tag{1}\\
\frac{dA^{-1}}{dx} &= -A^{-1}\frac{dA}{dx}A^{-1}.\tag{2}
\end{align}
Therefore
$$
\frac{d\log|A|}{dx}
=\frac1{|A|}\frac{d|A|}{dx}
=\operatorname{tr}\left(A^{-1} \frac{dA}{dx}\right)
=-\operatorname{tr}\left(\frac{dA^{-1}}{dx}A\right).
$$
