3
$\begingroup$

Recently, I was trying to take the following derivative $$ \dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V} $$ I was referring to matrix cookbook to solve it, where I found several useful equations:

Equation (59) says: $$ \dfrac{\partial Y^{-1}}{\partial x} = -Y^{-1}\dfrac{\partial Y}{\partial x}Y^{-1} $$ so, I think I have: $$ \dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V^{-1}} = -(X^TV^{-1}X)^{-1} X^TX(X^TV^{-1}X)^{-1} $$ and $$ \dfrac{\partial V^{-1}}{\partial V} = -V^{-1}V^{-1} $$ According to the chain rule, it should be: $$ \dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V} =\dfrac{\partial (X^TV^{-1}X)^{-1}}{\partial V^{-1}}\dfrac{\partial V^{-1}}{\partial V} = ((X^TV^{-1}X)^{-1} X^TX(X^TV^{-1}X)^{-1})^T V^{-1}V^{-1} $$

However, I met one problem. $V$ is a matrix of size $(n, n)$ and $X$ is a matrix of size $(n, m)$. Then, the first half of the chain rule is of size of $(m, m)$, while the second half of the chain rule is of size $(n, n)$.

Please help me figure out what goes wrong.

Thanks ahead.

$\endgroup$
1
  • $\begingroup$ ur first line isn't an equation $\endgroup$
    – tired
    Nov 23, 2015 at 21:35

2 Answers 2

Reset to default
1
$\begingroup$

Let's define some intermediate variables $$\eqalign{ P &= V^{-1} \cr M &= X^TPX \cr F &= M^{-1} \cr }$$ whose differentials are $$\eqalign{ dP &= -V^{-1}\,dV\,V^{-1} \cr dM &= X^T\,dP\,X \cr dF &= -M^{-1}\,dM\,M^{-1} \cr }$$ That last differential is the one we're interested in, so let's successively substitute variables until we get back to $V$ $$\eqalign{ dF &= -M^{-1}\,dM\,M^{-1} \cr &= -M^{-1}\,(X^T\,dP\,X)\,M^{-1} \cr &= -M^{-1}\,X^T\,(-V^{-1}\,dV\,V^{-1})\,X\,M^{-1} \cr &= M^{-1}\,X^T\,V^{-1}\,dV\,V^{-1}\,X\,M^{-1} \cr &= F\,X^T\,V^{-1}\,dV\,V^{-1}\,X\,F \cr }$$ At this point, let's follow the prescription of Magnus & Neudecker for dealing with matrix-by-matrix derivatives, and vectorize both sides $$\eqalign{ d{\rm vec}(F) &= (V^{-1}\,X\,F)^T\otimes(F\,X^T\,V^{-1})\,d{\rm vec}(V) \cr }$$ Which can be rearranged to the conventional looking result $$\eqalign{ \frac{\partial f}{\partial v} &= (V^{-1}\,X\,F)^T\otimes(F\,X^T\,V^{-1}) \cr }$$

$\endgroup$
3
  • $\begingroup$ Thank you very much. Do you mind say something about how $\otimes$ is defined for two matrices? I am asking because numpy implementation only unfolds two matrices back to vectors and do a outer product of vectors, which contradicts an example I found here $\endgroup$
    – user134111
    Nov 25, 2015 at 0:36
  • $\begingroup$ Also, while thank you for your solution, which saves me. I am also looking for why my original solution is incorrect, so that I won't run into similar problem in the future. I sense that my understanding of chain rule for matrix calculus is incorrect. Do you mind help me figure that out? $\endgroup$
    – user134111
    Nov 25, 2015 at 0:51
  • $\begingroup$ The $\otimes$ symbol stands for the Kronecker product. The Kronecker-vec identity is a well-known application. As for the chain rule, I do not use it for matrix-by-matrix derivatives, because it requires the use of double-dot products, and intermediate quantities which are 4th order tensors. If you insist on using the chain rule, you should switch to index notation (and the Einstein summation convention). $\endgroup$
    – lynn
    Nov 25, 2015 at 1:03
0
$\begingroup$

I don´t see how equation (59) should help. $V$ does not depend on $X$.

If $X$ is a square matrix then $\left(X^TV^{-1}X \right)^{-1}=X^{-1}V\left(X^{-1}\right)^T$

Let $X^{-1}=A$. We get $AVA^T$

You can see here (page 9) that $$\frac{\partial AVA^T}{\partial V}= AA^T$$

$\endgroup$
5
  • $\begingroup$ Thanks. @calculus. But the inverse of X does not necessarily exist since it is not square. $\endgroup$
    – user134111
    Nov 24, 2015 at 1:43
  • $\begingroup$ @HaohanWang That´s true. I added this condition. I don´t have a better idea. I´m right that $V$ doesn´t depend on $X$ ? $\endgroup$
    – callculus42
    Nov 24, 2015 at 1:51
  • $\begingroup$ Yes, $V$ does not depend on $X$ $\endgroup$
    – user134111
    Nov 24, 2015 at 2:36
  • $\begingroup$ @HaohanWang As a said, in this case i don´t think that equation 59 can be used here. $\endgroup$
    – callculus42
    Nov 24, 2015 at 2:45
  • $\begingroup$ do you mind elaborate why? If we see $Y$ as $X^{T}V^{-1}X$, $Y$ depends on $V$, isn't it? $\endgroup$
    – user134111
    Nov 24, 2015 at 22:27

You must log in to answer this question.