0
$\begingroup$

Let

$f(X) = \text{tr}(\log(X)\cdot A)$,

where $\log(X)$ is the matrix logrithm of matrix $X$, both $X$ and $A$ are $m\times m$ symmetric positive definite (SPD) matrices. I was wondering what is $\frac{\partial f}{\partial X}$?

My solution:

Let $Z= \log(X)$, and I assume (am not quite sure) that $dZ = Z^{-1}dX$. Then we have

$df = \text{tr}(X^{-1}dXA) = \text{tr}(AX^{-1}dX)$,

which gives

$\frac{\partial f}{\partial X} = X^{-1}A$.

It that correct?

Addition

What if

$f(X) = \text{tr}([\log(X)]^2A)$?

Using the similar method, let $Z= [\log(X)]^2$, and I assume (am still not quite sure) that $dZ = 2ZX^{-1}dX$. Then we have

$df = \text{tr}(2ZX^{-1}dXA) = 2\text{tr}(AZX^{-1}dX)$,

which gives

$\frac{\partial f}{\partial X} = 2X^{-1}ZA$.

$\endgroup$
3
  • $\begingroup$ Your formulas are correct only if $A$ commutes with $X$. If that's not the case, then there are no nice simple solutions. Positive definiteness doesn't really matter, but commutivity does. $\endgroup$
    – lynn
    Commented Dec 6, 2018 at 22:36
  • $\begingroup$ Thanks for the comments! Could please tell me why commutativity matters in this case? $\endgroup$ Commented Dec 6, 2018 at 22:48
  • $\begingroup$ Because of the trace operator, I thought commutativity is not a problem. $\endgroup$ Commented Dec 6, 2018 at 22:53

1 Answer 1

2
$\begingroup$

$ \def\h{\odot} \def\o{{\tt1}} \def\bR#1{\big(#1\big)} \def\BR#1{\Big[#1\Big]} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\Diag#1{\op{Diag}\LR{#1}} \def\tr#1{\op{tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\m#1{\left[\begin{array}{r}#1\end{array}\right]} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} \def\fracLR#1#2{\LR{\frac{#1}{#2}}} \def\gradLR#1#2{\LR{\grad{#1}{#2}}} $ Here is a numerical counter-example to your first formula using random, non-commuting, $2\times 2$, SPD matrices. $$ A = \m{ 29 & 57 \\ 57 & 117 \\ },\,\,\,\, X = \m{ 20 & 44 \\ 44 & 100 \\ },\,\,\,\, dX = \m{ 4 & 5 \\ 5 & 18 \\ }\times 10^{-4} $$ Let's estimate $df$ using your formula versus a direct calculation. $$\eqalign{ f(X) &= \tr{\log(X)\cdot A} \\ f(X+dX)-f(X) &= 0.002849328 \\ \tr{AX^{-1}dX} &= 0.000975000 \\ \Delta &= 65.8\% \cr }$$ To follow up on @lynn's comment, let's see what happens if the matrices commute. The simplest way to ensure that is to set $A=X$ and repeat the calculation. $$\eqalign{ f(X+dX)-f(X) &= 0.00219992 \\ \tr{AX^{-1}dX}= \tr{dX} &= 0.00220000 \ \Delta &= 0.004\% \\ \\ }$$

Update

Here is a simple non-numerical example of what goes wrong when the matrices don't commute. $$\eqalign{ f &= \tr{X^3A} \\ df &= \tr{X^2\,dX\,A + X\,dX\,XA + dX\,X^2A} \\ \grad fX &= X^2A + XAX + AX^2 \\ }$$ A rather ugly and complicated result for such a simple function. However if $(A,X)$ commute, then you can combine terms to obtain $$\grad fX = 3X^2A$$

Now imagine expanding a matrix function as a Taylor series, and then taking its derivative term-by-term. Each term $X^k$ will explode into $k$ distinct terms and you'll end up with a horrible mess. But you could do it.

However, for the $\log$ function, you can't even write down a Taylor series because it's singular at zero.

Update 2

For SPD matrices, the $\sf Daleckii$-$\sf Krein\;Theorem$ yields a closed-form solution.

First, calculate the Eigenvalue Decomposition $$\eqalign{ \def\b{\beta} X &= QBQ^T,\qquad I=Q^TQ,\;B=\Diag{\b_k} \\ }$$ Applying a generic function $h(z)$ and its derivative $h'(x)$ to $X$ yields $$\eqalign{ H_x &= h(X),\quad &H_b = h(B) \qiq &H_x = Q\,H_b\,Q^T \\ H_x' &= h'(X),\quad &H_b' = h'(B) \qiq &H_x' = Q\,H_b'\,Q^T \\ }$$ Since $B$ is diagonal, its functions are very easy to evaluate.

According to the DK Theorem the differential of this function is $$\eqalign{ Z &= \zeta(BJ-JB), \quad R = \fracLR{H_bJ-JH_b+ZH_b'}{BJ-JB+Z} \\ dH_x &= Q\BR{R\h\LR{Q^TdX\,Q}}Q^T \\ }$$ where $J$ is the all-ones matrix, $\LR{F\h G}$ denotes elementwise multiplication, $\fracLR FG$ denotes elementwise division, and $\zeta$ is an elementwise zero-indicator function, i.e. $$\eqalign{ \zeta\!\LR{\m{-2 & \c0 & 3\\9 & -5 & \c0}} = \m{0 & \c\o & 0\\0 & 0 & \c\o} }$$ In the current problem $$\eqalign{ h(x) &= \log(x), \qquad h'(x) = x^{-1} \\ }$$ The final piece of notation that we need is the Frobenius $(:)$ product $$\eqalign{ F:G &= \sum_{i=1}^m\sum_{j=1}^n F_{ij}G_{ij} \;=\; \tr{F^TG} \\ G:G &= \frob{G}^2 \\ F:G &= G:F \;=\; G^T:F^T \\ \LR{PQ}:G &= P:\LR{GQ^T} \;=\; Q:\LR{P^TG} \\ \LR{E\h F}:G &= E:\LR{F\h G} \\ }$$ Putting this all together $$\eqalign{ f &= A:H_x \\ df &= A:dH_x \\ &= A:\LR{Q\BR{R\h\LR{Q^TdX\,Q}}Q^T} \\ &= \LR{Q\BR{R\h\LR{Q^TAQ}}Q^T}:dX \\ \grad fX &= Q\BR{R\h\LR{Q^TAQ}}Q^T \\ }$$

$\endgroup$
5
  • $\begingroup$ Thanks for the comments! The counter-example is convincing, but why my derivation is wrong? Is there any analytical solution for the derivative of $f$? Thanks! $\endgroup$ Commented Dec 7, 2018 at 19:49
  • $\begingroup$ Thanks for the non-numerical example! I guess I got your point. But I still don't understand why we cannot use tricks from matrix calculus for this problem (e.g., math.stackexchange.com/questions/2793467/…)? In other words, when can we use these tricks and when can't we? $\endgroup$ Commented Dec 8, 2018 at 2:43
  • $\begingroup$ You can use the "trace trick" only when everything inside the trace is a function of $X$ and only $X$. You cannot have another matrix like $A$ inside of the trace. You cannot even have a "related" matrix like $(X^T, X^H, X^*)$ inside of the trace. The identity matrix allowed since it's technically a polynomial of $X$, i.e. $I=X^0$. And scalar coefficients are also permitted. $\endgroup$
    – greg
    Commented Dec 8, 2018 at 5:54
  • $\begingroup$ There is an important special case which I omitted from my previous comment. The linear function can accomodate a matrix coefficient, e.g. $$d\,{\rm tr}(AX) = {\rm tr}(A\,dX)$$ $\endgroup$
    – greg
    Commented Dec 8, 2018 at 6:10
  • $\begingroup$ I see... In words, we can use the trace trick for $\text{tr}(\log(X)), \text{tr}((\log(X))^2)$, but not for $\text{tr}(A\log(X))$, right? Thanks! $\endgroup$ Commented Dec 10, 2018 at 0:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .