2
$\begingroup$

Suppose I have a scalar function $L(W)$ $L:\mathbb{R}^{n \times n} \rightarrow \mathbb{R}$.

I would like to know the directional derivative $\frac{dL}{dW}$ with respect to some vector $v$.

Background: $L$ is the loss of a neural network parametrized by $W$, e.g. the entries $w_{ij}$ are the connection weights of, say, a vanilla RNN. At initialisation $w_{ij}$ ~ $\mathcal{N}_{(0,1)}$, so the eigenvalues are uniformly distributed on the unit disk. Now, suppose I train the network for a duration $t$. This introduces some perturbation $\Delta W=W_t-W_0$, causing some eigenvalues to separate from the bulk. Say, I have $k$ such outliers $\lambda_1, \ldots, \lambda_k$, associated with eigenvectors $v_1, \ldots, v_k$. Now, during the forward pass, the dynamics unfolds in a subspace spanned by the $v_i$, as all other eigenvalues have modulus below unity. What I would like to measure is basically how much of the gradient at some later time $t'$, $\frac{dL}{dW_t'}$ is contained in the subspace spanned by the $v_i$, i.e. I want some scalar output, which I guess is somewhat analogous to the directional derivative of some vector-valued $f(x)$ function which tells you how much the $f$ slopes in the direction of some vector $v$. In effect this measures the correlation between the gradient of $f$ and $v$. So basically I would like to do something similar only for a function defined over matrices, if that even makes sense.

Perhaps there is something really obvious that I am missing. If not I'll be happy to elaborate and show some of my ansatzes.

$\endgroup$
4
  • 2
    $\begingroup$ From your definition ${\cal L}({\pmb W})$ is a scalar function, like, for example ${\cal L}({\pmb W})={\pmb a}^\intercal {\pmb W} {\pmb a}$. For a scalar function of a vector, $\phi({\pmb v})$, the directional derivative along a vector $\pmb u$ is defined as ${\pmb u}^\intercal\frac{\partial \phi}{\partial {\pmb v}}$. Since $\frac{\partial {\cal L}}{\partial {\pmb W}}$ is a matrix how would you define the directional derivative in this case? $\endgroup$
    – Ted Black
    Commented Sep 4 at 19:23
  • $\begingroup$ Thank you. You are completely right, I edited that now. Indeed, $\mathcal{L}$ has somewhat the functional form you adduce. In the particular example above, using the squared loss, the $\mathcal{L}(W) = [\hat{z}-z](I - W)^{-1} w_{out}w_{in}^T (I - W)^{-1}$. $\endgroup$
    – blueslimr
    Commented Sep 4 at 21:51
  • $\begingroup$ Your title and the content of your post don't really match. The title doesn't make much sense as Ted notes. To answer your actual question, I think a reasonable way of doing this would be to compute the directional derivative in the direction of an orthogonal projector onto $\mathrm{span}\{v_1,\dots,v_k\}$. Up to rescaling I think this is equivalent to computing the principal cosine between the span of $v_i$'s and the column space of the gradient. $\endgroup$
    – whpowell96
    Commented Sep 4 at 21:53
  • $\begingroup$ Okay, that was part of my thinking as well. What if I did an SVD of my matrix $\hat{\Delta W} \approx U\Sigma V^T$, where $\Sigma \in \mathbb{R}^{k \times k}$, i.e $\hat{\Delta W}$ is the best k-rank approximation of the matrix $\Delta W$. Would simply computing the directional derivative $tr(\frac{\partial \mathcal{L}}{\partial W} \hat{\Delta W})$ yield the desired result? $\endgroup$
    – blueslimr
    Commented Sep 5 at 8:46

0

Reset to default

You must log in to answer this question.