# a practical question about matrix derivative with inverse and chain rule: dimension mismatch

Recently, I was trying to take the following derivative $$\dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V}$$ I was referring to matrix cookbook to solve it, where I found several useful equations:

Equation (59) says: $$\dfrac{\partial Y^{-1}}{\partial x} = -Y^{-1}\dfrac{\partial Y}{\partial x}Y^{-1}$$ so, I think I have: $$\dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V^{-1}} = -(X^TV^{-1}X)^{-1} X^TX(X^TV^{-1}X)^{-1}$$ and $$\dfrac{\partial V^{-1}}{\partial V} = -V^{-1}V^{-1}$$ According to the chain rule, it should be: $$\dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V} =\dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V^{-1}}\dfrac{\partial V^{-1}}{\partial V} = ((X^TV^{-1}X)^{-1} X^TX(X^TV^{-1}X)^{-1})^T V^{-1}V^{-1}$$

However, I met one problem. $V$ is a matrix of size $(n, n)$ and $X$ is a matrix of size $(n, m)$. Then, the first half of the chain rule is of size of $(m, m)$, while the second half of the chain rule is of size $(n, n)$.

• ur first line isn't an equation Nov 23, 2015 at 21:35

Let's define some intermediate variables \eqalign{ P &= V^{-1} \cr M &= X^TPX \cr F &= M^{-1} \cr } whose differentials are \eqalign{ dP &= -V^{-1}\,dV\,V^{-1} \cr dM &= X^T\,dP\,X \cr dF &= -M^{-1}\,dM\,M^{-1} \cr } That last differential is the one we're interested in, so let's successively substitute variables until we get back to $V$ \eqalign{ dF &= -M^{-1}\,dM\,M^{-1} \cr &= -M^{-1}\,(X^T\,dP\,X)\,M^{-1} \cr &= -M^{-1}\,X^T\,(-V^{-1}\,dV\,V^{-1})\,X\,M^{-1} \cr &= M^{-1}\,X^T\,V^{-1}\,dV\,V^{-1}\,X\,M^{-1} \cr &= F\,X^T\,V^{-1}\,dV\,V^{-1}\,X\,F \cr } At this point, let's follow the prescription of Magnus & Neudecker for dealing with matrix-by-matrix derivatives, and vectorize both sides \eqalign{ d{\rm vec}(F) &= (V^{-1}\,X\,F)^T\otimes(F\,X^T\,V^{-1})\,d{\rm vec}(V) \cr } Which can be rearranged to the conventional looking result \eqalign{ \frac{\partial f}{\partial v} &= (V^{-1}\,X\,F)^T\otimes(F\,X^T\,V^{-1}) \cr }

• Thank you very much. Do you mind say something about how $\otimes$ is defined for two matrices? I am asking because numpy implementation only unfolds two matrices back to vectors and do a outer product of vectors, which contradicts an example I found here
– user134111
Nov 25, 2015 at 0:36
• Also, while thank you for your solution, which saves me. I am also looking for why my original solution is incorrect, so that I won't run into similar problem in the future. I sense that my understanding of chain rule for matrix calculus is incorrect. Do you mind help me figure that out?
– user134111
Nov 25, 2015 at 0:51
• The $\otimes$ symbol stands for the Kronecker product. The Kronecker-vec identity is a well-known application. As for the chain rule, I do not use it for matrix-by-matrix derivatives, because it requires the use of double-dot products, and intermediate quantities which are 4th order tensors. If you insist on using the chain rule, you should switch to index notation (and the Einstein summation convention).
– lynn
Nov 25, 2015 at 1:03

I don´t see how equation (59) should help. $V$ does not depend on $X$.

If $X$ is a square matrix then $\left(X^TV^{-1}X \right)^{-1}=X^{-1}V\left(X^{-1}\right)^T$

Let $X^{-1}=A$. We get $AVA^T$

You can see here (page 9) that $$\frac{\partial AVA^T}{\partial V}= AA^T$$

• Thanks. @calculus. But the inverse of X does not necessarily exist since it is not square.
– user134111
Nov 24, 2015 at 1:43
• @HaohanWang That´s true. I added this condition. I don´t have a better idea. I´m right that $V$ doesn´t depend on $X$ ? Nov 24, 2015 at 1:51
• Yes, $V$ does not depend on $X$
– user134111
Nov 24, 2015 at 2:36
• @HaohanWang As a said, in this case i don´t think that equation 59 can be used here. Nov 24, 2015 at 2:45
• do you mind elaborate why? If we see $Y$ as $X^{T}V^{-1}X$, $Y$ depends on $V$, isn't it?
– user134111
Nov 24, 2015 at 22:27