# Derivative of a function w.r.t. matrix $K$ and scalar $\sigma^2$

Suppose that $K$ is an $r\times r$ symmetric and positive definite matrix, and that $D=(\sigma^2+\sigma^2_{\epsilon})^{-1} I$ is a $n\times n$ matrix with scalars $\sigma^2>0$ and $\sigma^2_{\epsilon}>0$. Let $\Sigma= SKS'+D^{-1}$ be an $n\times n$ positive definite matrix with $r<n$. Now we are interested in finding the second derivative of function $f$ w.r.t. unknown parameters $K$ and $\sigma^2$ respectively: $$f(K, \sigma^2) = \log |\Sigma| + z' \Sigma^{-1} z,$$ where $z$ is an $n \times 1$ vector.

Notice that $$|\Sigma|= |K^{-1} + S'DS|\cdot |K| \cdot |D^{-1}|,$$ and $$\Sigma^{-1} = D-DS(K^{-1}+S'DS)^{-1}S' D.$$ And we have $\frac{\partial^2 \log|K|}{\partial K^2} = - K^{-2}$.

Any suggestions will be appreciated!

Thanks,

• perhaps the $\det\exp A=\exp \mathrm{Tr} A$ formula could be useful here in order to make the $\log \lvert\Sigma\rvert$ term more "attractive", i.e., in loose terms $\log \lvert \Sigma\rvert=\mathrm{Tr} \log \Sigma$
– ewcz
Commented Sep 28, 2015 at 21:40

You can rearrange the function, and write it in terms of the Frobenius (:) product as \eqalign{ f &= \log(\det(\Sigma)) + z^T\Sigma^{-1}z \cr &= {\rm tr}(\log(\Sigma)) + zz^T:\Sigma^{-1} \cr } Now take the differential of the function and make successive variable substitutions until you get it terms of the variables that you are interested in \eqalign{ df &= \Sigma^{-T}:d\Sigma + zz^T:d\Sigma^{-1} \cr &= \Sigma^{-1}:d\Sigma - zz^T:\Sigma^{-1}\,d\Sigma\,\Sigma^{-1} \cr &= \Sigma^{-1}:d\Sigma - \Sigma^{-1}zz^T\Sigma^{-1}:d\Sigma \cr &= (\Sigma^{-1} - \Sigma^{-1}zz^T\Sigma^{-1}):d\Sigma \cr &= B:d\Sigma \cr &= B:(S\,dK\,S^T + dD^{-1}) \cr &= B:(S\,dK\,S^T + Id\sigma^2) \cr &= S^TBS:dK + B:Id\sigma^2 \cr &= S^TBS:dK + {\rm tr}(B)\,d\sigma^2 \cr } Now setting $dK=0$ yields the derivative with respect to $\sigma^2$ \eqalign{ \frac{\partial f}{\partial\sigma^2} &= {\rm tr}(B) \cr &= {\rm tr}(\Sigma^{-1} - \Sigma^{-1}zz^T\Sigma^{-1}) \cr } And setting $d\sigma^2=0$ yields \eqalign{ \frac{\partial f}{\partial K} &= S^TBS \cr &= S^T(\Sigma^{-1} - \Sigma^{-1}zz^T\Sigma^{-1})S \cr }
• How can we find the second derivatives such as $\frac{\partial^2 f}{\partial K \partial \sigma^2}$ and $\frac{\partial^2 f}{\partial K^2}$? I am not familiar with matrix derivatives. It would be very nice if you can give some references on it. Commented Oct 2, 2015 at 17:24