0
$\begingroup$

I'm so confused what the "error delta" means in backpropagation network. Is is same as Gradient? so Derivative means error?

Michael Nielslon said "partial derivative Cost function with respect to partial derivtive of each neuron is error delta"

so, "Ratio" is error? and we try to calculate derivative to zero?

I feel like "error" means "cost"... but error is not cost? mmm???

$\endgroup$
0
$\begingroup$

No. $\delta$ is not the same as the gradient. Note that $\delta$ has one value per neuron, while the gradient has as many components as weights and biases are in your network (so, a ton of them)

partial derivative Cost function with respect to partial derivtive of each neuron is error delta

This means $\delta$ measures how much the cost function would change after a tiny chance in the output of each of the neurons. But note that the model parameters we want to optimize are not the outputs of neurons, but rather the weights and biases. So, $\delta$ is somewhat related to the gradient (and we use it to get the actual gradient) but they are not the same thing.

Derivatives are not the same as errors. Error is measured by the cost function. Partial derivatives of the cost function indicate how that cost function would react to small changes in the network. In principle, we define, for each observation $x$, the cost function of that observation as $C_x(\vec{w},\vec{b})$, so, a function perfectly defined in terms of weigths and biases. But, thanks to the chain rule, we can still measure thing like $\frac{\partial C}{\partial a_i^L}$ where $a_i^L$ represents the output of the $i$th neuron in the $L$th layer. Derivaitves like those are what we use to get $\delta$, which is really just a trick to get the gradient in an efficient way

More info at http://neuralnetworksanddeeplearning.com/chap2.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.