4
$\begingroup$

Chapter 8.7 of the Deep Learning Book says "Suppose our cost function has put a gradient of 1 on $\hat y$ …" Does it mean the derivative of the cost function w.r.t $\hat y$ is equal to 1?

For example, my model has 2 hidden layers and 1 output layer, so

$\hat y=xw_1w_2w_3$

Suppose the model is using mean squared error as loss function, then the loss for n data points is defined as

$loss{(\hat y, y)} = {\dfrac {1}{n}}\sum _{i=1}^{n}(y_{i}-{\hat y_{i}})^{2}$

where both $\hat y$ is the output vector the model predicts and $y$ denotes the corresponding ground truth label.

Does "put a gradient of 1 on $\hat y$" mean the following?

$\dfrac{\partial \ loss{(\hat y, y)}}{\partial \hat y} = 1$

The part cited above comes from Page 314 of the book

enter image description here

$\endgroup$
7
  • $\begingroup$ Could you maybe add the page number to make it easier to find the context? $\endgroup$
    – csch2
    Commented Aug 13, 2021 at 4:15
  • $\begingroup$ @JJJohn This is for you, the bounty placer. It seems like the assertion is correct. The cost function, evaluated at $\hat{y}$, has a derivative of $1$. Since we want to decrease the cost, we must proceed (back-propagate) in the direction where the cost decreases i.e. lowering the weights. This is why the analysis considers the quantity $w \to w-\epsilon \mathbf{g}$. The calculation for the first order term is given by equation 4.9 (in the same book) and the point i/s that the second and third order effects that appear from expansion can be too large, hence batch renormalization is considered. $\endgroup$ Commented Aug 27, 2021 at 4:25
  • $\begingroup$ Note that the derivative isn't $1$ for all values of $\hat{y}$ (if so, then it's a stupid notion of cost). Instead, we have fixed some $w_i$ while initializing the learning , so the $\hat{y}$ corresponding to these parameters is the point, at which the derivative of the cost function is $1$. We are obviously looking for a point where the derivative becomes zero, so we proceed in the direction opposite to the local increase of the cost function , in this case the derivative is positive so we go backward $\endgroup$ Commented Aug 27, 2021 at 4:27
  • $\begingroup$ "Does "put a gradient of 1 on y^" mean the following?" Yes $\endgroup$
    – Bananach
    Commented Aug 27, 2021 at 7:24
  • $\begingroup$ @TeresaLisbon Thank you. Would you consider moving your comments to answer? $\endgroup$
    – JJJohn
    Commented Aug 28, 2021 at 11:30

1 Answer 1

1
+50
$\begingroup$

The cost function evaluated at $\hat{y}$, has a derivative of $1$. Since we want to decrease the cost, we must proceed (i.e. back-propagate) in the direction where the cost decreases i.e. by lowering the weights.

This is why the analysis considers the quantity $\mathbf{w} \to \mathbf{w} - \epsilon \mathbf {g}$. The calculation for the first order term is given by equations 4.5 (in the same book, page 86) which is : $$ x' = x - \epsilon \nabla_xf(x) $$

and the point is that if one sees equations $4.9,4.10$ from the same book, then one sees the second order effects of gradient descent as well. The formula given above, when matched with the situation we face and the iteration $w \to w - \epsilon g$, shows that our guess is correct.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .