# Directional derivative of a matrix with respect to a vector

Suppose I have a scalar function $$L(W)$$ $$L:\mathbb{R}^{n \times n} \rightarrow \mathbb{R}$$.

I would like to know the directional derivative $$\frac{dL}{dW}$$ with respect to some vector $$v$$.

Background: $$L$$ is the loss of a neural network parametrized by $$W$$, e.g. the entries $$w_{ij}$$ are the connection weights of, say, a vanilla RNN. At initialisation $$w_{ij}$$ ~ $$\mathcal{N}_{(0,1)}$$, so the eigenvalues are uniformly distributed on the unit disk. Now, suppose I train the network for a duration $$t$$. This introduces some perturbation $$\Delta W=W_t-W_0$$, causing some eigenvalues to separate from the bulk. Say, I have $$k$$ such outliers $$\lambda_1, \ldots, \lambda_k$$, associated with eigenvectors $$v_1, \ldots, v_k$$. Now, during the forward pass, the dynamics unfolds in a subspace spanned by the $$v_i$$, as all other eigenvalues have modulus below unity. What I would like to measure is basically how much of the gradient at some later time $$t'$$, $$\frac{dL}{dW_t'}$$ is contained in the subspace spanned by the $$v_i$$, i.e. I want some scalar output, which I guess is somewhat analogous to the directional derivative of some vector-valued $$f(x)$$ function which tells you how much the $$f$$ slopes in the direction of some vector $$v$$. In effect this measures the correlation between the gradient of $$f$$ and $$v$$. So basically I would like to do something similar only for a function defined over matrices, if that even makes sense.

Perhaps there is something really obvious that I am missing. If not I'll be happy to elaborate and show some of my ansatzes.

• From your definition ${\cal L}({\pmb W})$ is a scalar function, like, for example ${\cal L}({\pmb W})={\pmb a}^\intercal {\pmb W} {\pmb a}$. For a scalar function of a vector, $\phi({\pmb v})$, the directional derivative along a vector $\pmb u$ is defined as ${\pmb u}^\intercal\frac{\partial \phi}{\partial {\pmb v}}$. Since $\frac{\partial {\cal L}}{\partial {\pmb W}}$ is a matrix how would you define the directional derivative in this case? Commented Sep 4 at 19:23
• Thank you. You are completely right, I edited that now. Indeed, $\mathcal{L}$ has somewhat the functional form you adduce. In the particular example above, using the squared loss, the $\mathcal{L}(W) = [\hat{z}-z](I - W)^{-1} w_{out}w_{in}^T (I - W)^{-1}$. Commented Sep 4 at 21:51
• Your title and the content of your post don't really match. The title doesn't make much sense as Ted notes. To answer your actual question, I think a reasonable way of doing this would be to compute the directional derivative in the direction of an orthogonal projector onto $\mathrm{span}\{v_1,\dots,v_k\}$. Up to rescaling I think this is equivalent to computing the principal cosine between the span of $v_i$'s and the column space of the gradient. Commented Sep 4 at 21:53
• Okay, that was part of my thinking as well. What if I did an SVD of my matrix $\hat{\Delta W} \approx U\Sigma V^T$, where $\Sigma \in \mathbb{R}^{k \times k}$, i.e $\hat{\Delta W}$ is the best k-rank approximation of the matrix $\Delta W$. Would simply computing the directional derivative $tr(\frac{\partial \mathcal{L}}{\partial W} \hat{\Delta W})$ yield the desired result? Commented Sep 5 at 8:46