2
$\begingroup$

Let $A\in\mathbb{R}^{n\times m}$, $n\geq m$, be a full column rank matrix, and consider the function \begin{align} f&\colon \mathbb{R}^{n\times n} \to \mathbb{R}^{n\times n}\\ & X\mapsto A (A^\top X A)^{-1} A^\top, \end{align} where $\bullet^\top$ denotes transposition.

Assuming that $(A^\top X A)^{-1}$ exists, I'm interested in the computation of the Jacobian matrix of $f$, i.e. $$\tag{1}\label{a} \mathbf{J}[f] = \left[\frac{\partial f(X)}{\partial X_{ij}}\right]\in\mathbb{R}^{n^2\times n^2}. $$

I know that there exists a closed form expressions for the Jacobian of the inverse, namely $\mathbf{J}[X^{-1}]=-(X^{-\top} \otimes X^{-1})$ (see e.g. here, page 5). Hence, I wonder whether a similar closed-form expression can be derived for \eqref{a}.

Thanks in advance.

$\endgroup$
2
$\begingroup$

Given $\mathrm A \in \mathbb R^{n \times m}$, matrix-valued function $\mathrm F : \mathbb R^{n \times n} \to \mathbb R^{n \times n}$ is defined as follows

$$\mathrm F (\mathrm X) := \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top}$$

Hence,

$$\mathrm F (\mathrm X + h \mathrm V) = \mathrm A \left( \mathrm A^{\top} (\mathrm X + h \mathrm V) \mathrm A \right)^{-1} \mathrm A^{\top} = \cdots = \mathrm F (\mathrm X) - h \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \mathrm V \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top}$$

Thus, the directional derivative of $\mathrm F$ in the direction of $\mathrm V$ at $\mathrm X$ is the matrix-valued function

$$- \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \mathrm V \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top}$$

Making $\mathrm V = \mathrm e_i \mathrm e_j^{\top}$, we obtain

$$\partial_{x_{ij}} \mathrm F (\mathrm X) = - \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \mathrm e_i \mathrm e_j^{\top} \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} = \color{blue}{- \mathrm F (\mathrm X) \, \mathrm e_i \mathrm e_j^{\top} \mathrm F (\mathrm X)}$$

which is a multiple of the outer product of the $i$-th column and $j$-th row of $\mathrm F (\mathrm X)$.

Vectorizing the directional derivative, we obtain

$$\mbox{vec} \left( - \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \mathrm V \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \right) = \color{blue}{- \left( \mathrm A \left( \mathrm A^{\top} \mathrm X^{\top} \mathrm A \right)^{-1} \mathrm A^{\top} \otimes \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \right)} \mbox{vec} (\mathrm V)$$

$\endgroup$
  • $\begingroup$ Thank you for your answer! Before accepting it, I would like to ask you a (perhaps, silly) clarification. I’m a bit confused on the “right” notion of Jacobian for a matrix-valued function, namely between the classical one (your first blue formula) and the vectorized one (your second blue formula). For instance, if I would like to check whether or not the Jacobian is non-singular for a given positive definite $X$, which formula do I have to consider? $\endgroup$ – Ludwig Feb 11 '17 at 9:46
  • 1
    $\begingroup$ @Jacquard Take my formula and make $\mathrm A = \mathrm I_n$. You then obtain $- (\mathrm X^{-\top} \otimes \mathrm X^{-1})$, which is the formula in the question and also in Fackler's notes. Hence, I suppose that is the "right" formula. $\endgroup$ – Rodrigo de Azevedo Feb 11 '17 at 10:49
  • 1
    $\begingroup$ @Jacquard If $- \left( \mathrm A \left( \mathrm A^{\top} \mathrm X^{\top} \mathrm A \right)^{-1} \mathrm A^{\top} \otimes \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \right)$ is singular, then there is at least one direction $\mathrm V$ for which the directional derivative of $\mathrm F$ is zero. That would be one critical point. $\endgroup$ – Rodrigo de Azevedo Feb 11 '17 at 10:55
  • 1
    $\begingroup$ @Jacquard The directional derivative is $- \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top} \mathrm V \mathrm A \left( \mathrm A^{\top} \mathrm X \mathrm A \right)^{-1} \mathrm A^{\top}$. If the rows or columns of $\mathrm V$ are in the left null space of $\mathrm A$, then the directional derivative is zero. $\endgroup$ – Rodrigo de Azevedo Feb 11 '17 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.