Taking a derivative of a normal function is easy, but taking a derivative of an element of a vector has not been explained in my textbooks. The vector function is relatively simple, but it involves a summation of its elements.
$$l\left(y,\hat{y}\right)=\log \sum _{k=1}^q\:\exp\left(o_k\right)-\sum _{j=1}^qy_jo_j$$ $$ \partial_{o_j}l\left(y,\hat{y}\right)=\frac{\exp\left(o_j\right)}{\displaystyle\sum _{k=1}^q\exp\left(o_k\right)}-y_j$$
Notice that o, y and ŷ are vectors. Important note: those vectors have $q$ components. Also $ŷ = softmax(o)$ and the sum of y components equals to 1. What I want to examine is the second row here - where the answer is given for my derivative.
The second component, which is $-y_j$ should be the easiest. Since we are actually looking for a partial derivative, I suppose we treat $-y_j$ as a coefficient of $o_j$, so it's not surprising we see it in our formula. However, the summation has gone and I really wonder how does that happen?
As for the first component, well I know for a fact that $\ln \left(e^x\right)=x$ and $\frac{d}{dx}\left(\log \left(e^x\right)\right)=1$. But since we have a log of sum of $e$ to the something... that of course results to something entirely different, which is $\frac{e^{o_i}}{\sum _{k=1}^qe^{o_k}\:}$. So once again, I really wonder how does that happen?
An explanation of what's going on here would be highly appreciated.
