Is there a close form for $$ \frac{\partial}{\partial \mathbf{X}} \mathbf{X} (\mathbf{X}^\top \mathbf{X})^{-1/2}, $$ where $\mathbf{A}^{-1/2}$ is the inverse square root of a matrix (eg. inverse of the Cholesky decomposition)?
2
-
$\begingroup$ you can use the chain rule. The derivative of the square root of a positive definite matrix can be read off from it's power series, and the derivative of the matrix inverse can also be read off from it's power series. $\endgroup$– MasonCommented Dec 2, 2023 at 4:58
-
$\begingroup$ Notice that if you use the functional calculus square root (rather than cholesky), then this matrix is equal to the product of left and right singular vector matrices, $UV^T$. So, you could probably approach the problem via perturbation theory for the SVD. Here is a related post on mathoverflow, though there are no answers there: mathoverflow.net/questions/409651/… $\endgroup$– Nick AlgerCommented Dec 4, 2023 at 1:01
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
$ \def\K{K^{(m,n)}} \def\B{B^{-1}} \def\k{\otimes} \def\kp{\oplus} \def\Im{I_m} \def\In{I_n} \def\R#1{{\mathbb R}^{#1}} \def\BR#1{\left[#1\right]} \def\bR#1{\Big[#1\Big]} \def\lR#1{\Big(#1\Big)} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} $Assume that $X\in\R{m\times n}$ and for typing convenience define $B\in\R{n\times n}$ as $$\eqalign{ &B = B^T = \LR{X^TX}^{1/2} \qiq b = \vc B \qquad \qquad \quad }$$ Differentiating wrt $X$ yields $$\eqalign{ &B^2 = X^TX \\ &dB\,B+B\:dB = dX^TX + X^TdX \\ &\LR{B\k\In+\In\k B}db = \bR{\LR{X^T\k\In}\K + \LR{\In\k X^T}}\,dx \\ &db = \lR{B\kp B}^{-1}\,\bR{\LR{X^T\k\In}\K + \LR{\In\k X^T}}\,dx \\ &db = {G}\:dx \qiq G = \grad{\vc B}{\vc X} \\ }$$ where $\k$ is the Kronecker product, $\kp$ is the Kronecker sum, $\K$ is the Commutation Matrix associated with the vectorization of $X^T$ and $G$ is the gradient after vectorization.
The preceding gradient can be used to calculate the desired gradient $$\eqalign{ F &= X\LR{X^TX}^{-1/2} \\ &= XB^{-1} \\ dF &= dX\,\B - X\B\,dB\:\B \\ &= \Im\:dX\,\B - F\,dB\:\B \\ df &= \LR{\B\k\Im}dx - \LR{\B\k F}db \\ &= \bR{\LR{\B\k\Im} - \LR{\B\k F}G}\,dx \\ \grad{\vc F}{\vc X} &= \;\LR{\B\k\Im} - \LR{\B\k F}G \\ }$$
-
$\begingroup$ I wonder if there is any sort of cancellation here that would make the result surprisingly stable? Since $X(X^TX)^{-1/2}=UV^T$, where $U$ and $V$ are the left and right singular vector matrices, this seems like the SVD version of the spectral projector. In the eigenvalue case we know that perturbations to the spectral projectors are stable even when perturbations of the eigenvalue problem are not (see, e.g., here: mathoverflow.net/a/370014/24119). $\endgroup$ Commented Dec 4, 2023 at 1:48