Derivative of matrix inversion function? Let's say I have a function $f$ which maps any invertible $n\times n$ matrix to its inverse. How do I calculate the derivative of this function?
 A: To calculate the directional derivatives of the map, at a fixed matrix $A$:
Suppose you change entry $i,j$ of your matrix $A$ by adding a small $t$ (small enough that the result is still invertible). Let $E_{i,j}$ be the matrix with a $1$ in the $i,j$th position and $0$ elsewhere. 
Note that 
$$
(A + tE_{i,j})(A + tE_{i,j})^{-1} = I
$$
so on one hand, the derivative with respect to $t$ is zero (as the identity is constant), on the other hand, the derivative can be written
$$
\frac{d(A + tE_{i,j})}{dt}(A + tE_{i,j})^{-1} + (A + tE_{i,j})\frac{d(A + tE_{i,j})^{-1}}{dt} = 0
$$
so 
$$
\frac{d(A + tE_{i,j})^{-1}}{dt} = -(A + tE_{i,j})^{-1}\frac{d(A + tE_{i,j})}{dt}(A + tE_{i,j})^{-1}\\ = -(A + tE_{i,j})^{-1}E_{i,j}(A + tE_{i,j})^{-1}
$$
A: $$(X+\delta)^{-1} = [X\cdot ( I + X^{-1} \cdot \delta)]^{-1}= ( I + X^{-1} \cdot \delta)^{-1}\cdot X^{-1} = \\
=(I - X^{-1} \delta + X^{-1} \delta X^{-1} \delta - \ldots ) \cdot X^{-1}= X^{-1} - X^{-1} \delta X^{-1} +  X^{-1} \delta X^{-1} \delta X^{-1} - \ldots 
$$
so the linear part of the variation is $-X^{-1} \delta X^{-1}$.
We also have the Taylor series expansion at $X$.
