1
$\begingroup$

TL;DR: Given a feed-forward neural network with $l$ layers, what is the partial derivative of the slope of a linear regression on the average layer-wise activations with respect to some weight $w_{i,j}$?

Specifically, I have some neural network with, say three layers, i.e.

$f(x) := \sigma (W_3 \sigma(W_2 \sigma(W_1 x)))$ with a three dimensional output, say, with size of 3 for each layer (to keep things simple).

For a given input, each neuron will have an outputted activation level, $z_k^l$ (neuron $k$ at $l$-th layer). We can define the average activation for a layer as $\frac{\sum_k (z_k^l)^2}{n_l}$, where $n_l$ denotes the number of neurons in layer $l$, and get a vector of average activations with each entry corresponding to a layer. Denote this vector as $a = [\frac{\sum_k (z_k^l)^2}{n_l}]_{\forall l}$.

We can perform a linear regression on this vector. As an example, let $\beta = [\beta_0, \beta_1]^T$, $C = [\mathbf{1}, a]$. Then, $\beta_1$ is the slope of a linear regression $a = C \beta$ on the activations. Then, $\beta_1$ is equal to the second entry of $C^\dagger a$.

My question is, how can I differentiate this function with respect to a weight $w_{i,j}$?

I have tried it, but it gets messy given you are working with the pseudoinverse. Explicitly expressing it in terms of the neuronal activations seems cumbersome.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.