0
$\begingroup$

Taking a derivative of a normal function is easy, but taking a derivative of an element of a vector has not been explained in my textbooks. The vector function is relatively simple, but it involves a summation of its elements.

$$l\left(y,\hat{y}\right)=\log \sum _{k=1}^q\:\exp\left(o_k\right)-\sum _{j=1}^qy_jo_j$$ $$ \partial_{o_j}l\left(y,\hat{y}\right)=\frac{\exp\left(o_j\right)}{\displaystyle\sum _{k=1}^q\exp\left(o_k\right)}-y_j$$

Notice that o, y and ŷ are vectors. Important note: those vectors have $q$ components. Also $ŷ = softmax(o)$ and the sum of y components equals to 1. What I want to examine is the second row here - where the answer is given for my derivative.

The second component, which is $-y_j$ should be the easiest. Since we are actually looking for a partial derivative, I suppose we treat $-y_j$ as a coefficient of $o_j$, so it's not surprising we see it in our formula. However, the summation has gone and I really wonder how does that happen?

As for the first component, well I know for a fact that $\ln \left(e^x\right)=x$ and $\frac{d}{dx}\left(\log \left(e^x\right)\right)=1$. But since we have a log of sum of $e$ to the something... that of course results to something entirely different, which is $\frac{e^{o_i}}{\sum _{k=1}^qe^{o_k}\:}$. So once again, I really wonder how does that happen?

An explanation of what's going on here would be highly appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

Ok, it becomes self explanatory after considering the simplest case with a summation over a 2 row vector:

$$l\left(y,ŷ\right)=log\left(e^{o_i}+e^{o_j}\right)-\left(y_io_i+y_jo_j\right)$$

$$\frac{\partial l}{\partial o_j}=\frac{e^{o_j}}{\sum _{k∈\left\{i,j\right\}}^{ }e^{o_k}\:}-y_j$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .