TL;DR: Given a feed-forward neural network with $l$ layers, what is the partial derivative of the slope of a linear regression on the average layer-wise activations with respect to some weight $w_{i,j}$?

Specifically, I have some neural network with, say three layers, i.e.

$f(x) := \sigma (W_3 \sigma(W_2 \sigma(W_1 x)))$ with a three dimensional output, say, with size of 3 for each layer (to keep things simple).

For a given input, each neuron will have an outputted activation level, $z_k^l$ (neuron $k$ at $l$-th layer). We can define the average activation for a layer as $\frac{\sum_k (z_k^l)^2}{n_l}$, where $n_l$ denotes the number of neurons in layer $l$, and get a vector of average activations with each entry corresponding to a layer. Denote this vector as $a = [\frac{\sum_k (z_k^l)^2}{n_l}]_{\forall l}$.

We can perform a linear regression on this vector. As an example, let $\beta = [\beta_0, \beta_1]^T$, $C = [\mathbf{1}, a]$. Then, $\beta_1$ is the slope of a linear regression $a = C \beta$ on the activations. Then, $\beta_1$ is equal to the second entry of $C^\dagger a$.

My question is, how can I differentiate this function with respect to a weight $w_{i,j}$?

I have tried it, but it gets messy given you are working with the pseudoinverse. Explicitly expressing it in terms of the neuronal activations seems cumbersome.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.