# How to Derive Softmax Function

Can someone explain step by step how to to find the derivative of this softmax loss function/equation.

$$L_i=-log(\frac{e^{f_{y_{i}}}}{\sum_j e^{f_j}}) = -f_{y_i} + log(\sum_j e^{f_j})$$

where: $$f = w_j*x_i$$ let:

$$p = \frac{e^{f_{y_{i}}}}{\sum_j e^{f_j}}$$

The code shows that the derivative of $L_i$ when $j = y_i$ is:

$$(p-1) * x_i$$

and when $j \neq y_i$ the derivative is:

$$p * x_i$$

It seems related to this this post, where the OP says the derivative of:

$$p_j = \frac{e^{o_j}}{\sum_k e^{o_k}}$$

is:

$$\frac{\partial p_j}{\partial o_i} = p_i(1 - p_i),\quad i = j$$

But I couldn't figure it out. I'm used to doing derivatives wrt to variables, but not familiar with doing derivatives wrt to indxes.

• I had a similar question and got an amazing answer: stackoverflow.com/questions/37790990/… Commented Jun 13, 2016 at 14:10
• Cool, thanks @roshini I'll check that out!
– Khon
Commented Jun 14, 2016 at 17:20

We have a softmax-based loss function component given by: $$L_i=-log\left(\frac{e^{f_{y_i}}}{\sum_{j=0}^ne^{f_j}}\right)$$

Where:

1. Indexed exponent $f$ is a vector of scores obtained during classification
2. Index $y_i$ is proper label's index where $y$ is column vector of all proper labels for training examples and $i$ is example's index

Objective is to find: $$\frac{\partial L_i}{\partial f_k}$$

Let's break down $L_i$ into 2 separate expressions of a loss function component: $$L_i=-log(p_{y_i})$$

And vector of normalized probabilities:

$$p_k=\frac{e^{f_{k}}}{\sum_{j=0}^ne^{f_j}}$$

Let's substitute sum:

$$\sigma=\sum_{j=0}^ne^{f_j}$$

For $k={y_i}$ using quotient rule:

$$\frac{\partial p_k}{\partial f_{y_i}} = \frac{e^{f_k}\sigma-e^{2f_k}}{\sigma^2}$$

For $k\neq{y_i}$ during derivation $e^{f_k}$ is treated as constant:

$$\frac{\partial p_k}{\partial f_{y_i}} = \frac{-e^{f_k}e^{f_{y_i}}}{\sigma^2}$$

Going further:

$$\frac{\partial L_i}{\partial p_k}=-\left(\frac {1}{p_{y_i}}\right)$$

Using chain rule for derivation:

$$\frac{\partial L_i}{\partial f_k}=-\left(\frac {1}{\frac{e^{f_{k}}}{\sigma}}\right)\frac{\partial p_k}{\partial f_{y_i}}=-\left(\frac {\sigma}{{e^{f_{k}}}}\right)\frac{\partial p_k}{\partial f_{y_i}}$$

Considering $k$ and $y_i$, for $k=y_j$ after simplifications:

$$\frac{\partial L_i}{\partial f_k}=\frac{e^{f_k}-\sigma}{\sigma}=\frac{e^{f_k}}{\sigma}-1=p_k-1$$

And for $k\neq y_j$:

$$\frac{\partial L_i}{\partial f_k}=\frac{e^{f_k}}{\sigma}=p_k$$

These two equations can be combined using Kronecker delta:

$$\frac{\partial L_i}{\partial f_k}=p_k-\delta_{ky_i}$$