1
$\begingroup$

After studying backpropagation for neural networks, I have a question: why can't we use full derivatives for backpropagation? I understand why partial derivatives work in backpropagation. However I wonder why we cannot (or should not) use full derivatives.

$\endgroup$
1
  • 1
    $\begingroup$ If you edit in a derivation of backpropagation you've seen, you can pinpoint the first use of partial derivatives you feel warrants a detailed explanation, but the answer will be an application of multivariable chain rules of the form$$\frac{\partial f}{\partial t}=\sum_i\frac{\partial f}{\partial x_i}\frac{\partial x_i}{\partial t}.$$ $\endgroup$
    – J.G.
    May 4, 2021 at 16:34

1 Answer 1

1
$\begingroup$

It is because you ultimately want to find "in which direction" you should change the network's parameters (biases and weights), one by one, in order to minimize the loss. So you look at every parameter and tweak it slightly from its value and see if the loss increases or decreases: this is equivalent to computing

$$\frac{\partial \mathcal{L}}{\partial b^{(l)}_j} \hspace{0.5cm} or \hspace{0.5cm} \frac{\partial \mathcal{L}}{\partial w^{(l)}_{ij}}$$

where $(l)$ is the layer the bias or weight refers to.

To compute these quantities you use backpropagation (which is just the chain rule applied to neural networks) and this involves partial derivatives.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .