I am following Andrew NG's Machine Learning course on Coursera.
The cost function without regularization used in the Neural network course is:
$J(\theta) = \frac{1}{m} \sum ^{m}_{i=1}\sum ^{K}_{k=1} [-y_{k}^{(i)}log((h_{\theta}(x^{(i)}))_{k}) -(1-y_{k}^{(i)})log(1-(h_{\theta}(x^{(i)}))_{k})]$
, where $m$ is the number of examples, $K$ is the number of classes, $J(\theta)$ is the cost function, $x^{(i)}$ is the i-th training example, $\theta$ are the weight matrices and $h_{\theta}(x^{(i)})$ is the prediction of the neural network for the i'th training example.
I understand intuitively that the backpropagation error associated with the last layer(h) is h-y. Nevertheless, I want to be able to prove this formally.
For simplicity, I considered m = K = 1:
$J(\theta) = -y \log(h_{\theta}) - (1-y) \log(1-h_{\theta})$
and tried to prove this to myself on paper but wasn't able to.
Neural Network Definition:
This neural network has 3 layers. (1 input, 1 hidden, 1 output).
It uses the sigmoid activation function,
$\sigma(z) = \frac{1}{1+e^{-z}}$. The input is $x$.
Input layer: $a^{(1)} = x$. (add bias $a_{0}^{(1)}$).
Hidden Layer: $z^{(2)} = \Theta^{(1)}a^{(1)}$ , $a^{2} = \sigma(z^{(2)})$, (add bias $a_{0}^{(2)}$).
Output layer: $z^{(3)} = \Theta^{(2)}a^{(2)}$ , $a^{3} = \sigma(z^{(3)}) = h_{\theta}(x)$.
During backpropagation, $\delta^{(3)}$ is the error associated with the output layer.
Question:
- Why is it that:
$\delta^{(3)} = h_{\theta} - y$ ?
- Shouldn't:
$\delta^{(3)} = \frac{\partial {J}} {\partial {h_{\theta}}}$ ?