I'm reading a popular book on neural networks, here's the link. http://neuralnetworksanddeeplearning.com/chap3.html In the excerpt below $C$ denotes the cost function, $b$ - bias, $w_i$ - $i$-th connection weight, $x_i$ - $i$-th component of the input vector, $a$ - output activation for a single neuron with multiple inputs and single output.

[...] the cost $C=Cx$ for a single training example $x$ would satisfy

$\frac{\partial C}{\partial w_i } = x_j(a - y) $ (71)

$\frac{\partial C}{\partial b } = (a - y) $ (72)

If we could choose the cost function to make these equations true, then they would capture in a simple way the intuition that the greater the initial error, the faster the neuron learns. They'd also eliminate the problem of a learning slowdown. In fact, starting from these equations we'll now show that it's possible to derive the form of the cross-entropy, simply by following our mathematical noses. To see this, note that from the chain rule we have

$\frac{\partial C}{\partial b} = \frac{\partial C}{\partial a} \sigma'(z) $

Then they integrate this equation and get the following expression for C:

$ C=−[ylna+(1−y)ln(1−a)]+constant $

This is the contribution to the cost from a single training example, x. To get the full cost function we must average over training examples, obtaining $ C=−\frac{1}{n} \sum_{x} [ylna+(1−y)ln(1−a)]+constant, $ where the constant here is the average of the individual constants for each training example. And so we see that Equations (71) and (72) uniquely determine the form of the cross-entropy, up to an overall constant term. The cross-entropy isn't something that was miraculously pulled out of thin air. Rather, it's something that we could have discovered in a simple and natural way.

I don't see how the equation 71 is involved in this derivation, it seems to me that only the second one (72) was used. I'm feeling that it's something obvious but I'm really stuck and can't read any further :)


Recall that $a = \sigma(z)$ and $z = \sum_j w_j x_j + b$. Then (71) and (72) imply respectively: $$ \partial_{w_j} C = x_j (a - y) \;\;\;\&\;\;\; \partial_{b} C = (a - y) $$ To get (73), we remember that $C(y,a) = C(y,z) = C(y,w,x,b)$; i.e., $C$ is a function of $b$ through $a$ (which relates them through $z$). This is not related to (71) or (72). Thus, by the chain rule $$ \partial_b C = \partial_z C \underbrace{\partial_b z}_1 = \partial_a C \underbrace{\partial_z a}_{\partial_z \sigma(z)} = \sigma'(z) \partial_a C = a(1-a)\partial_a C \tag{74} $$ where the definition of $\sigma$ implies that $\sigma'(z)=\sigma(z)(1 - \sigma(z)) = a(1-a)$. Thus, using (72) and (74), we get $$ \partial_a C = \frac{a - y}{a(1-a)} $$ which can be integrated like $$ \int \frac{a - y}{a(1-a)} da = \int \left( \frac{1}{1-a} - \frac{y}{a(1-a)} \right) da = (y-1)\log(1-a) - y\log(a) + \gamma $$ where the constant $\gamma$ (which doesn't matter for optimization) can be determined by e.g. setting $C(y=0.5,a=0.5) = -\log 0.5 $.

Ok, so we never used (71). We didn't need to. What this implies is that (72) is already a sufficient constraint (combined with (1) the form of $\sigma$ [which we used to get (74)] and (2) the relation between $b$ and $a$ [which determined the chain rule form in (74)]).

Suppose $a = \sigma(z) = \sigma\left( f(w,x) + b \right) $. Thus (71) no longer holds, in general. By specifying (72), we say nothing about $f$. How can it be that (72) determines $C$ without constraining $f$?

Well, we view $C$ is a function of $y$ and the parameters $w,b$. Since $\sigma$ is fixed, by specifying (72), we are constraining the derivative of $C$ wrt $b$ to be some specific function. However, the form of $z$ and the properties of $\sigma$ mean that this actually constrains the derivative wrt $a$ (as seen in equation (74), we are fixing the derivative wrt $a$ to be some specific function $g(y,a) = y-a$). But $C$ is only a function of $a$ and $y$, and for a given input $y$ is fixed. Thus, constraining $\partial_a C$ to a be a smooth function completely determines $C$ on its own.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.