Calculating the matrix derivative $\frac{\partial}{\partial X}||CX^{-1}B||_F^2$

For matrices $$C, X$$, and $$B$$, I know that $$\frac{\partial}{\partial X}||CXB||_F^2 = 2C^TCXBB^T$$, and that $$\partial X^{-1} = -X^{-1}(\partial X) X^{-1}$$. However, I am unable to combine these results to calculate $$\frac{\partial}{\partial X}||CX^{-1}B||_F^2$$.

• And what happens if you try to apply the chain rule? Do you get something that you know is wrong, or it just looks very complicated and you are not convinced you are correct? (Or, are you having a confusion with the chain rule itself?) Sep 16 '20 at 5:32
• I'm not aware of how the chain rule generalizes to matrix derivatives. Sep 16 '20 at 7:36
• I see. But you have seen matrix derivatives, is that right? So I would not need to introduce it if I were to write an answer. Sep 16 '20 at 8:05
• Yes, I understand what a matrix derivative is. Sep 16 '20 at 8:12
• Thanks, I will write an answer(sorry for the delay, I was busy in work). Sep 16 '20 at 16:33

For ease of typing, let's define $$F = CX^{-1}B$$

Its differential is given by: $$dF = (dC)X^{-1}B + C(dX^{-1})B + CX^{-1}(dB)$$

We have $$dA=0$$, $$dB=0$$, and we can calculate $$dX^{-1}$$ as follows :

$$$$\begin{split} X^{-1}X & = I \\ \implies dX^{-1}X + X^{-1}dX & = dI = 0 \\ \implies dX^{-1}X & = - X^{-1}dX \\ \implies dX^{-1} & = - X^{-1}(dX)X^{-1} \\ \end{split}$$$$

Back to our expression, we have,
$$$$\begin{split} Y &= ||F||_F^2 \\ & = \text{Tr}(F^TF) = F:F \\ \implies dY & = dF:F + F:dF \\ & = F:dF + F:dF \\ & = 2F:dF \\ & = 2CX^{-1}B:C(dX^{-1})B \\ & = 2CX^{-1}B:-CX^{-1}(dX)X^{-1}B \\ & = -2(CX^{-1})^TCX^{-1}B:(dX)X^{-1}B \\ & = -2(CX^{-1})^TCX^{-1}B(X^{-1}B)^T:dX \\ \end{split}$$$$

Finally, we get: $$$$\begin{split} \frac{\partial (CX^{-1}B)}{\partial X} &= -2(CX^{-1})^TCX^{-1}B(X^{-1}B)^T\\ &= -2(X^{-1})^TC^TCX^{-1}BB^T(X^{-1})^T \end{split}$$$$

Define a new matrix variable \eqalign{ Y &= X^{-1} \qquad\implies\qquad dY &= -Y\,dX\,Y \\ } Use the gradient, which you already know, to write the differential of the function in terms of $$Y$$ \eqalign{ \phi &= \|CYB\|^2 \\ d\phi &= 2\Big(C^TCYBB^T\Big):dY \\ } Then perform a change of variables from $$Y\to X$$ \eqalign{ d\phi &= 2\Big(C^TCYBB^T\Big):\Big(-Y\,dX\,Y\Big) \\ &= -2\Big(Y^TC^TCYBB^TY^T\Big):dX \\ &= -2\Big((X^{-1})^TC^TCX^{-1}BB^T(X^{-1})^T\Big):dX \\ \frac{\partial \phi}{\partial X} &= -2(X^{-1})^TC^TCX^{-1}BB^T(X^{-1})^T \\ }

In the above, a colon is used as a product notation for the trace, i.e. \eqalign{A:B = {\rm Tr}(A^TB) = {\rm Tr}(B^TA) = B:A} The terms in such a product can be rearranged in a number of equivalent ways, e.g. \eqalign{ A:B &= A^T:B^T \\ A:BC &= B^TA:C = AC^T:B \\ } due to the properties of the trace function.

As you have discovered, the chain rule is difficult to apply in Matrix Calculus. It often requires the calculation of intermediate quantities which are third and fourth order tensors.

The beauty of the differential approach is that the differential of a matrix acts like a matrix. In particular, it obeys all of the rules of matrix algebra.

The function $$X \to \|CX^{-1}B\|_F^2$$ is a composition of the functions $$g(X)= X^{-1}$$ followed by $$f(Y) = \|CYB\|_F^2$$.

The crux of the multidimensional derivative is the following : for $$f : \mathbb R^n \to \mathbb R^m$$ differentiable everywhere, the derivative $$f'$$ is an association such that for any $$y \in \mathbb R^n$$, $$f'(y) : \mathbb R^n \to \mathbb R^m$$ is a linear transformation (hence a matrix), which is given by $$f'(y)v = \lim_{t \to 0}\frac{f(y+tv) - f(y)}{t}$$

Therefore, noting that $$M_{n \times n}$$ may also be equated to a space $$\mathbb R^{n^2}$$ in our explanation above, we have $$f : M_{n \times n} \to \mathbb R$$, hence $$f'(Y)$$ for any $$Y \in M_{n \times n}$$ is a linear transformation from $$M_{n \times \mathbb n} \to \mathbb R$$. According to what you have derived, we have : $$[f'(Y)]M = 2C^TCYBB^TM$$

Similarly, $$g(X) = X^{-1}$$ is a function from $$M_{n \times n} \to M_{n \times n}$$, so $$g'(Y)$$ for any $$Y \in M_{n \times n}$$ is a linear transformation from $$M_{n \times n} \to M_{n \times n}$$ given by : $$[g'(Y)]N = -Y^{-1}NY^{-1}$$

The chain rule tells you that the derivative of $$f \circ g$$ is : $$[f \circ g]'(X) = f'(g(X)) \cdot g'(X)$$

where $$\cdot$$ indicates matrix multiplication (or composition of the linear transformations which are the derivatives).

In particular, for any $$M$$ we have : $$[[f \circ g]'(X)]M = f'(g(X)) [[g'(X)]M]$$

Note that if $$f \circ g$$ is well defined then this matrix multiplication will also go through without a dimension problem.

In our case, we have $$[g'(X)]M = -X^{-1}MX^{-1}$$, and finally, we get that $$[[f \circ g]'(X)](M) = f'(g(X))[g'(X)M] = -2C^TCX^{-1}BB^TX^{-1}M X^{-1}$$

is how the derivative acts as a linear map at each $$M \in M_{n \times n}$$.

• @Powerspawn Was there a doubt you had on the question? Because I saw that you had made a comment and then it got deleted before I could read it. Sep 17 '20 at 2:47
• Thank you for this. I do have another question, it's my understanding that as a matrix, $\frac{\partial f(X)}{\partial X}(M) = [ \frac{\partial f}{\partial x_{ji}}]_{ij}(M)$. Is it possible to represent $-2C^TCX^{-1}BB^TX^{-1}M X^{-1}$ in the form $AM$ for some matrix $A$? Sep 17 '20 at 2:49
• @Powerspawn The expression $-2C^TCX^{-1}BB^TX^{-1}MX^{-1}$ is going to be a real number for each matrix $M$. If I multiply $M$ by a matrix $A$, the quantity $AM$ is not going to be a real number, so I don't think this is possible : in fact, the general non-commutativity of matrices means that you can't bring that last $X^{-1}$ to the left of the $M$. Sep 17 '20 at 2:55
• @Powerspawn I should also add something : the derivative $f'(X) = 2C^CXBB^T$ is actually somewhat incorrect. The correct expression is $[f'(X)](M) = 2tr(C^TCXBB^TM)$, but we leave out the trace if the argument is a scalar. We can't do that in general, of course : accordingly I will modify the above answer. Sep 17 '20 at 3:06
• @Powerspawn That makes sense too, but in this case to match the dimensions, we must take the trace. Anyway, in that case I won't modify the answer ; the key point is the chain rule and the fact that linear operators on matrices won't always be right multiplication by a matrix but could end up being complicated. Sep 17 '20 at 3:21