Derivative concepts for matrix function. Matrix valued functions can be defined as power series expansions:
$$f(X) = \sum_{k=0}^Nc_kX^k$$
The derivative for scalar valued functions is usually defined as a limit:
$$f'(x) = \lim_{h\to 0} \frac{f(x+h)-f(x)}{h}$$
But what would the derivative of such a function for arguments $X$ being a matrix mean? What should the $h$ in the limits be replaced by if we replace scalar $x$ by matrix $X$? And what should we replace the division by $h$ with?
 A: We first recall that the derivative of a function $f:\mathbb{R} \to \mathbb{R}$ can be thought of as a linear map $L:\mathbb{R} \to \mathbb{R}$ so that
$$ \frac{\lvert f(x+h)-f(x)-L(h) \rvert}{\lvert h \rvert} \to 0 \quad \text{as } h \to 0. $$
It happens that all such maps are given by multiplication by a number, $L(h)=ah$, say, giving the idea that $f'(x)$ is a number. (Given this, we may now note this relation holds if and only if there is a  $\frac{f(x+h)-f(x)}{h} \to a$ as $h \to 0$, the usual quotient.)
The usual quotient doesn't work in more dimensions, because there is more than one direction to approach the point from. Moreover, we also have more linear maps than just multiplication by numbers. We therefore introduce the Fréchet derivative, which works on any complete normed vector space: a function $F:\mathbb{R}^n \to \mathbb{R}^m$ is called differentiable if there is a linear map $L:\mathbb{R}^n \to \mathbb{R}^m$ so that
$$ \frac{\lVert F(x+h)-F(x)-L(h) \rVert}{\lVert h \rVert} \to 0 \quad \text{as } h \to 0. $$
If this is the case, $L$ is called the derivative of $F$ at $x$, and is written various ways, of which $DF_x$ is one. If we think of $\mathbb{R}^n$ and $\mathbb{R}^m$ as containing vectors, this is the Jacobian.
Now, the space of $k\times l $ matrices $M_{k\times l}$ is a finite-dimensional normed vector space (and hence complete), with one norm being $\lVert A \rVert_F=\sqrt{\operatorname{tr}{(A^TA)}}$, for example. Suppose we have a function $F: M_{k \times l} \to M_{m \times n}$. Then the derivative will be a linear map $L: M_{k \times l} \to M_{m \times n}$ so that
$$ \frac{\lVert F(X+H)-F(H)-L(H) \rVert_F}{\lVert H \rVert_F} \to 0 \quad \text{as } H \to 0. $$

This is how to define a derivative on matrix functions. Actually calculating the derivative for a specific function is a rather different process, since matrix multiplication is not commutative. A simple example where the computation can be done directly is the squaring map, $S:X \mapsto X^2$. Then
$$ S(X+H)-S(X)=(X+H)^2-X^2 = XH+HX + H^2, $$
from which we find that $DS_X(H)= XH+HX $. One can produce similar expressions for higher powers. The inverse map $X \mapsto X^{-1}$ can be shown to have derivative $D(X^{-1})(H) = -X^{-1}HX^{-1}$, and so on. (We notice that in both cases the answer reduces to the scalar result if we assume the multiplication is commutative, which is plausible).
