1
$\begingroup$

This is a question that arose from watching 3blue1brown's video on the explanation of backpropagation in neural networks using calculus. I am familiar with the chain rule (for some context, I was taught these rules during my A-Level Maths course, which has now tragically been left unexamined due to the pandemic whose name shan't be said aloud), but there was one derivative that I was unsure of because it surely couldn't make sense in this context.

The equation was:

zL = wLaL-1 + bL

where L indicates the current layer of neurons (hence L-1 is the previous layer), w is a weight, a is the activation of some neuron, and b is the bias.

According to Grant in his video, taking the derivative of this gives you

d(zL) ⁄ d(wL) = aL-1

I understand that wL has its power reduced from 1 to 0, hence (wL)0 = 1, but what I don't understand is why bL seems to disappear - surely it is not a constant (even when looking with respect to wL) because changing the bias DOES affect the gradient of the cost function (I'm not sure if cost function was used by Grant in a way to explain this to viewers or if the cost function exists as a standardised way of calculating the scale of error produced by the neural network, so for better context of where I'm coming from as a beginner to machine learning, it is explained in the video I linked above). If it were a constant, or at least if it had some mathematical explanation as to why it behaves as a constant, I would certainly understand how it reduces to 0 ( bL is multiplied by (wL)0 = 1, and so taking the derivative of the whole equation would result in multiplying (wL)0 by 0 and decrementing its exponent, hence bL is multiplied by 0 and 'disappears'). But I can't understand how it behaves as such, since a change in bL must surely affect the gradient of the cost function, and not just shift the final cost function up or down by some value.

I hope this was all worded well, this is a very new subject area to me and I'm usually reluctant to post on stack exchange (either this maths stack exchange or stack overflow), as I tend to get responses which I find condescending and sometimes rather abrasive, but I figured that regardless I can still pull the relevant help from responses and try to make it make sense to me - it would be a disaster to risk not understanding a topic solely because the fear of asking a silly question hinders me. I'm happy to elaborate on any confusing/vague wording I may have included.

Thanks!

  • RWN
$\endgroup$

2 Answers 2

2
$\begingroup$

It's probably best to regard this by looking at total derivatives rather than partial derivatives - the equation $\frac{dz^L}{dw^L}=a^L$ is really referring to the partial derivative of $z^L$ with respect to $w^L$ holding $b^L$ constant - but this should really be written with the partial derivative symbol $\partial$ and is an instance where the notation entirely fails to convey the "holding $b^L$" constant part of the meaning.

It's somewhat better to just write this as: $$dz^L=dw^L\cdot a^{L-1} + db^L + w^L\cdot d a^{L-1}$$ which says "the change in the outputs of the $L^{th}$ layer is the changes in the weight matrix applied to the existing activations plus the change in the biases plus the weight matrix times the change in the prior layer's activation." You could write this out as partial derivatives, but, as you fell victim to, this doesn't convey the meaning quite as clearly and leaves the necessity of coming back for the biases separately (which leaves more room for error). This form is also nice because the backpropagation algorithm is, in this form, just "keep substituting in for $d a^{L-1}$ until you get to the bottom."

$\endgroup$
1
  • $\begingroup$ Ahhh, see I suppose I was never aware there existed partial and absolute derivatives, and so I wasn't quite aware of why Grant's notation (he used ∂ specifically, where instead I was using the typical Newtonian dy/dx notation) was significant (or perhaps it wasn't and I'm reading into things). Thank you for your response! This is all really interesting! $\endgroup$
    – RWN
    May 18, 2020 at 21:04
0
$\begingroup$

I suppose I posted this slightly prematurely before finishing the video - burning questions tend to drag me on an immediate and undeterred search for an answer.

The bias IS indeed important in calculating some component of the gradient vector (for the cost function, i.e. what must change to minimise error); however, it can be done separately to recognise its change. Treating it as a constant isn't necessarily disregarding the bias' significance, since it is revisited in a separate calculation as a separate component of the gradient vector.

Sorry guys for being so rushed!!

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .