Differntiating matrix functions $f : \mathbb R^{n\times m} \to \mathbb R^{p\times q}$ How would you differentiate matrix functions $f : \mathbb R^{n\times m} \to \mathbb R^{p\times q}$ like for example $f(X) = X^T \cdot X$? There are no directional derivatives in the usual sense, and also there exists no Jacobian matrix.
 A: To answer this question: the total derivative of your example function $f:\Bbb R^{m \times n} \to \Bbb R^{m \times m}$ at a matrix $X$ is the linear map $df(X) : \Bbb R^{m \times n} \to \Bbb R^{m \times m}$ given by
$$
df(X)(H) = H^TX + X^TH.
$$
In general, $f : \mathbb R^{n\times m} \to \mathbb R^{p\times q}$ would have a derivative $df(X) : \mathbb R^{n\times m} \to \mathbb R^{p\times q}$. With vectorization, we could think of $df(X)$ as a matrix by considering the matrix of the corresponding map from $\Bbb R^{mn} \to \Bbb R^{pq}$. Arguably, however, it is more natural to think of this object as a multilinear map.  In particular, $df(X)$ can be "naturally" identified with a type $(2,2)$ tensor.
Note that we can, in fact, make sense of the notion of a directional derivative. In particular: given a "direction" matrix $V$, we could say that
$$
d_Vf(X) = \lim_{h \to 0} \frac{f(X + hV) - f(X)}{h} = df(X)(V).
$$
Keep in mind that what we call a "unit vector" as far as directions are concerned ultimately depends on which matrix-norm we use for $\Bbb R^{m \times n}$; the most common norm in this context is the "Frobenius" norm $\|\cdot\|_F$.  Note that the directional derivative only gives a "slope" in the usual sense if the output of $f$ is a scalar.
If $f: \Bbb R^{n \times m} \to \Bbb R$, then the total derivative of $f$ can be expressed in the form 
$$
df(X)(H) = \operatorname{tr}\left(\left(\frac{\partial f}{\partial X}(X) \right)^TH\right),
$$ 
where $\frac{\partial f}{\partial X}$ denotes the derivative of $f$ in the so-called "denominator layout" notation. Note that this form of $\frac{\partial f}{\partial X}$ is analogous to the gradient of $f$ in that the directional derivative is the "dot-product" with $\frac{\partial f}{\partial X}$, and $\frac{\partial f}{\partial X}$ points in the "direction" of maximal increase.
