I am trying to derive the derivative of the trace of inverse of a matrix function (of X), i.e. $$f(X)=Tr\left((HXH^{H}+I)^{-1}\right) $$ where $H\in R^{n\times m}, X\in R^{m\times m}$.

So $HXH^{H}+I$ is invertible while $HXH^{H}$ is not. Any one know how to find $\frac{\partial f}{\partial X}$ ?

Thanks in advance!

  • $\begingroup$ Presumably, $H^H$ means the conjugate transpose of $H$. Is this so? $\endgroup$ – Omnomnomnom Sep 1 '15 at 17:18
  • $\begingroup$ Yes, you are right. $\endgroup$ – Mafen Sep 2 '15 at 3:48
  • $\begingroup$ I am used to the notation $H^*$, so this is what I used in my answer. $\endgroup$ – Omnomnomnom Sep 2 '15 at 3:50

Note: In the below, I used $X$ instead of $H$ by accident. Also, $X^*$ is my notation for $X^H$.

We can find this inverse by the chain rule. Note that $f = f_1 \circ f_2 \circ f_3$, where $$ f_1(A) = Tr(A)\\ f_2(A) = A^{-1}\\ f_3(A) = XAX^* + I $$ The derivatives of these functions are given by $$ [f_1'(A)](B) = Tr(B)\\ [f_2'(A)](B) = -A^{-1}BA^{-1}\\ [f_3'(A)](B) = XBX^*\\ $$ The chain rule tells us that $$ [(f_1 \circ f_2 \circ f_3)'(A)] = \\ [f_1'(f_2(f_3(A)))]\circ [f_2'(f_3(A))] \circ f_3'(A) $$ So, all together, that gives us $$ [f'(A)] = \\ (B \mapsto Tr[(XBX^* + I)^{-1}])\circ (B \mapsto -A^{-1}(XBX^* + I)A^{-1})\circ (B \mapsto XBX^*) =\\ (B \mapsto Tr[(XBX^* + I)^{-1}])\circ (B \mapsto -A^{-1}(X[XBX^*]X^* + I)A^{-1}) =\\ (B \mapsto Tr[(XBX^* + I)^{-1}])\circ (B \mapsto -A^{-1}(X^2BX^{*2} + I)A^{-1}) =\\ (B \mapsto Tr[(X[-A^{-1}(X^2BX^{*2} + I)A^{-1}]X^* + I)^{-1}]) $$ In other words, if $A$ is parameterized as a function of $t \in \Bbb R$, then we can write $$ \frac{d}{dt}f(A(t)) = Tr[(X[-A^{-1}(X^2\frac{dA}{dt}X^{*2} + I)A^{-1}]X^* + I)^{-1}] $$

Clarification of my derivatives:

By $[f'(A)](B)$, I mean that for every $A$, $f'(A)$ is a function on $B$. You are right to be confused. Let's look at an example.

Take $f(X) = Tr(X)$, and find its derivative my way. In fact, $f$ is linear, so this will be easy: we have $$ [f'(A)](B) = Tr(B) $$ Note that, in this case, $A$ doesn't pop up in the definition of $f'$. In other words, $f$ has a constant derivative. This makes sense because $f$ is linear. Another way of writing the above is to say that if $A(t)$ is a matrix-valued function on $t \in \Bbb R$, then $$ \frac{d}{dt}f(A(t)) = Tr \frac{dA}{dt} $$ Now, my understanding of your notation is that $$ \frac{\partial f}{\partial X}[i,j] = \frac{\partial f}{x_{ji}} $$ equivalently, define $A_{ij}(t) = A + t E_{ji}$ (where $E_{ji}$ is the matrix with $0$s everywhere except at the $j,i$ entry, where it has a $1$). Then we should have $$ \frac{\partial f}{\partial X}(A)[i,j] = \frac {d}{dt} f(A_{ij}(t)) $$ By my above statement, that means we can write $$ \frac{\partial f}{\partial X}(A)[i,j] = Tr(E_{ji}) = \begin{cases} 1 & i=j\\ 0 & i \neq j \end{cases} $$ In other words, $\frac {\partial f}{\partial X}$ is $I$, which is exactly what we expected.

  • $\begingroup$ Thanks for your quick and detailed answer! But I haven't quite got it. What do you mean by $[f^{\prime}_1 (A)](B)$? $\endgroup$ – Mafen Sep 2 '15 at 4:03
  • $\begingroup$ Well, this is the notion of derivatives that I'm used to: if $f:X \to Y$, then at each $A \in X$, $f'(A)$ is a linear function from $X$ to $Y$ (this outlook is used in Rudin, for example). Unfortunately, I am not familiar enough with your $\frac{\partial f}{\partial X}$ notation to work well with it. I do attempt to clarify things with my equation at the end, though, if that's any help. $\endgroup$ – Omnomnomnom Sep 2 '15 at 4:08
  • $\begingroup$ If you play around with the last equation, you can get the partial derivatives that you need for the matrix $\frac{\partial f}{\partial X}$, whatever that happens to look like. $\endgroup$ – Omnomnomnom Sep 2 '15 at 4:13
  • $\begingroup$ Do you mean $[f^{\prime}(A)](B)$ by substituting $B$ into $A$ in $f^{\prime}(A)$? According to what I know, $\frac{\partial Tr{A}}{\partial A}=I$ and $\frac{\partial Tr{AB}}{\partial A}=B$. (Sorry for my weakness in math.) $\endgroup$ – Mafen Sep 2 '15 at 4:14
  • $\begingroup$ Not quite. I'll add something to the end of my answer to explain better. $\endgroup$ – Omnomnomnom Sep 2 '15 at 4:19

I got an answer myself in a relative simple way: $$\frac{\partial f}{\partial X}=H^H((HXH^H+I)^{-2})^{H}H$$ through the chain rule. That is, let $U=HXH^H$ and $g(U)=Tr((U+I)^{-1})$, then $$\frac{\partial f}{\partial X_{i,j}}=Tr[(\frac{\partial g(U)}{\partial U})^H \frac{\partial U}{\partial X_{i,j}}]$$.Since we can easily get $\frac{\partial g(U)}{\partial U}=(U+I)^{-2}$ and $$U=\sum_i \sum_j X_{i,j} h_i h_j^H$$, we know $$\frac{\partial U}{\partial X_{i,j}}=h_i h_j^H$$.We thus have $$\frac{\partial f}{\partial X_{i,j}}=Tr[((HXH^H+I)^{-2})^{H}h_ih_j^{H}]=Tr[h_j^{H}((HXH^H+I)^{-2})^{H}h_i]=h_j^{H}((HXH^H+I)^{-2})^{H}h_i$$. Reorganizing them in a compact form we have the above result (hopefully there is no error). Actually the $X$ I am dealing with is diagonal, so the derivation is simpler than above. Thank you all for your attention and especially for @Omnomnomnom!

  • $\begingroup$ This looks like it could be correct. Of course, it helps to know the derivative of $g(U)$ in this form, and to have a chain rule that works with your notation. $\endgroup$ – Omnomnomnom Sep 2 '15 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.