0
$\begingroup$

I am following Andrew NG's Machine Learning course on Coursera.

The cost function without regularization used in the Neural network course is:

$J(\theta) = \frac{1}{m} \sum ^{m}_{i=1}\sum ^{K}_{k=1} [-y_{k}^{(i)}log((h_{\theta}(x^{(i)}))_{k}) -(1-y_{k}^{(i)})log(1-(h_{\theta}(x^{(i)}))_{k})]$

, where $m$ is the number of examples, $K$ is the number of classes, $J(\theta)$ is the cost function, $x^{(i)}$ is the i-th training example, $\theta$ are the weight matrices and $h_{\theta}(x^{(i)})$ is the prediction of the neural network for the i'th training example.

I understand intuitively that the backpropagation error associated with the last layer(h) is h-y. Nevertheless, I want to be able to prove this formally.

For simplicity, I considered m = K = 1:

$J(\theta) = -y \log(h_{\theta}) - (1-y) \log(1-h_{\theta})$

and tried to prove this to myself on paper but wasn't able to.

Neural Network Definition:

This neural network has 3 layers. (1 input, 1 hidden, 1 output).

It uses the sigmoid activation function,

$\sigma(z) = \frac{1}{1+e^{-z}}$. The input is $x$.

Input layer: $a^{(1)} = x$. (add bias $a_{0}^{(1)}$).

Hidden Layer: $z^{(2)} = \Theta^{(1)}a^{(1)}$ , $a^{2} = \sigma(z^{(2)})$, (add bias $a_{0}^{(2)}$).

Output layer: $z^{(3)} = \Theta^{(2)}a^{(2)}$ , $a^{3} = \sigma(z^{(3)}) = h_{\theta}(x)$.

During backpropagation, $\delta^{(3)}$ is the error associated with the output layer.

Question:

  1. Why is it that:

$\delta^{(3)} = h_{\theta} - y$ ?

  1. Shouldn't:

$\delta^{(3)} = \frac{\partial {J}} {\partial {h_{\theta}}}$ ?

$\endgroup$
8
  • $\begingroup$ It is unclear. I think you made some mistakes copying the formulas. $\endgroup$ – reuns Jun 15 '17 at 15:35
  • $\begingroup$ The formula is copied correctly, I double-checked. Maybe I misunderstood something, but given that it says that for each output unit k in layer 3(the output layer), for backpropagation I should set: $\delta_{k}^{(3)} = (h(\theta)_{k}^{(3)} - y_{k})$ , I concluded that this must be true because: $\frac{\partial J(\theta)}{\partial h_{\theta}} = h_{\theta} - y$ , when k=1. Was I wrong to assume this? $\endgroup$ – Roland Jun 15 '17 at 15:53
  • 1
    $\begingroup$ The notations are horrible. Let $z^{(i)} = h_\theta(x^{(i)})$ the output for the $i$th input $x^{(i)}$. What we want (to apply the gradient descent) is $\frac{\partial J}{\partial \theta_{lj}}$, and for this we look at $\frac{\partial J}{\partial z^{(i)}_j}$ and $\frac{\partial z^{(i)}_j}{\theta_{lj}}$. Also you didn't define your neural network (ie. $h_\theta$) only the objective function. $\endgroup$ – reuns Jun 15 '17 at 16:07
  • 1
    $\begingroup$ You meant $$h_\theta(1-h_\theta) \frac{\partial J}{\partial h_{\theta}} = h_{\theta}-y$$ $\endgroup$ – reuns Jun 15 '17 at 16:18
  • 1
    $\begingroup$ Related: link $\endgroup$ – user3658307 Jun 15 '17 at 17:01
0
$\begingroup$

First, since your cost function is using the binary cross-entropy error $\mathcal{H}$ with a sigmoid activation $\sigma$, you can see that: \begin{align} \frac{\partial J}{\partial h_\theta} &= \frac{1}{m}\sum_i\sum_k\frac{\partial }{\partial h_\theta}\mathcal{H}\left(y_k^{(i)},h_\theta(x^{(i)})_k\right) \\ &= \frac{1}{m}\sum_i\sum_k \left[ \frac{-y_k^{(i)}}{h_\theta(x^{(i)})_k} + \frac{1-y_k^{(i)}}{1-h_\theta(x^{(i)})_k} \right] \\ &= \frac{1}{m}\sum_i\sum_k \frac{h_\theta(x^{(i)})_k - y_k^{(i)}}{ h_\theta(x^{(i)})_k(1-h_\theta(x^{(i)})_k) } \end{align} Hence, for $m=K=1$, as a commenter notes $$ \frac{\partial J}{\partial h_\theta} = \frac{h_\theta - y}{ h_\theta(1-h_\theta) } $$ But this is not so useful, as it computes how the error changes as the final output changes. What you really want is how the cost changes as the weights $\theta^{(\ell)}_{ij}$ are varied, so you can do gradient descent on them. An intermediate calculation is to compute the variation with respect to the activation $ h_\theta=\sigma(z)$. Let the last layer be $s$. Then the output layer error is: \begin{align} \delta^{(s)}_j &= \frac{\partial J}{\partial z_j^{(s)}}\\ &= \frac{1}{m}\sum_i\sum_k \frac{\partial }{\partial z_j^{(s)}} \mathcal{H}\left(y_k^{(i)},h_\theta(x^{(i)})_k\right) \\ &= \frac{-1}{m}\sum_i\sum_k y_k^{(i)} \frac{1}{h_\theta(x^{(i)})_k}\frac{\partial h_\theta(x^{(i)})_k}{\partial z_j^{(s)}} + (1-y_k^{(i)})\frac{1}{1-h_\theta(x^{(i)})_k}\frac{\partial h_\theta(x^{(i)})_k}{\partial z_j^{(s)}} \\ &= \frac{-1}{m}\sum_i\sum_k [1-h_\theta(x^{(i)})_k]y_k^{(i)} - h_\theta(x^{(i)})_k[1-y_k^{(i)}]\\ &= \frac{1}{m}\sum_i\sum_k h_\theta(x^{(i)})_k -y_k^{(i)} \end{align} using the fact that $$ \frac{\partial h_\theta(x^{(i)})_k}{\partial z_j^{(s)}} = \sigma'(z_j^{(s)}) = \sigma(z_j^{(s)})[1-\sigma(z_j^{(s)})] = h_\theta(x^{(i)})_k[1-h_\theta(x^{(i)})_k] $$ So in the case that $m=K=1$ and $s=3$, we have: $$ \delta^{(3)} = h_\theta - y $$

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.