0
$\begingroup$

From https://en.wikipedia.org/wiki/Softmax_function

In mathematics, the softmax function, or normalized exponential function, is a generalization of the logistic function that "squashes" a $K$-dimensional vector $z$ of arbitrary real values to a $K$-dimensional vector $\sigma(z)$ of real values in the range $(0, 1)$ that add up to $1$. The function is given by

$$\sigma(z)_j=\frac{e^{z_j}}{\sum_{k=1}^Ke^{z_k}},\quad \text{for }j=1,...,K$$

I just can't understand why this softmax equation can keep result values sum up to 1.


From M. Winter's answer, I know zj =zk, so the answer is simple. I feel sometimes hard to understand the English style explanation of some equations.

This one using j and k is confused, which make I think ezk is the previous layer output, and ezj is current new output. There is many equations contain such writing, some time j and k is diferent , just say both j and k are K-dimensional is not enough.

And ,In my opinion:

For example: if input X= [1,2,3,4,1,2,3], assign each element as Xi (i from 0 to X length 6). Why not just calculate Xi / SUM(X1 +.... X6) for each element ? e seems meanless here, which also a factor make me think ezk and ezj are different.

$\endgroup$
0
2
$\begingroup$

Try computing the sum of the components of $\sigma(z)$, i.e. $\sum_{j=1}^K \sigma(z)_j$. You will see

\begin{align} \sum_{j=1}^K \sigma(z)_j &= \sum_{j=1}^K \frac{e^{z_j}}{\sum_{k=1}^Ke^{z_k}} = \frac{\sum_{j=1}^K e^{z_j}}{\sum_{k=1}^Ke^{z_k}}=1, \end{align}

because the sum is linear and you can pull out the other sum from the inside.

$\endgroup$
4
  • $\begingroup$ Using j and k is confused, which make I think ezk is the previous layer output, and ezj is current new output. There is many equations contain such writing, some time j and k is diferent , just say both j and k are K-dimensional is not enough. $\endgroup$ – Mithril May 6 '17 at 3:30
  • $\begingroup$ And for example, if input X= [1,2,3,4,1,2,3], assign each element as Xi (i from 0 to X length 6). Why not just calculate Xi / SUM(X1 +.... X6) for each element . e seems meanless, which also a factor make me think ezk and ezj are different. $\endgroup$ – Mithril May 6 '17 at 3:40
  • 1
    $\begingroup$ @Mithril Suppose $K=3$. Then $\frac{\sum_{j=1}^K e^{z_j}}{\sum_{k=1}^K e^{z_k}}=\frac{e^{z_1}+e^{z_2}+e^{z_3}}{e^{z_1}+e^{z_2}+e^{z_3}}=1$. The fact that you use different letters to index the two sums doesn't mean they have different values. I think in general you may be a bit confused about summation notation. $\endgroup$ – Ian May 6 '17 at 3:55
  • $\begingroup$ @Mithril The $e$ is there to make the answer positive. Without this, the $\sigma(z)_j$ could be a bunch of numbers, still adding up to 1, but also negative, like $-1000$ and $1001$. This is unwanted and you want results in $[0,1]$. And no matter what you plug in into $e^z$, the resuklt is always positive. So this ensures the $\sigma(z)_j>0$ part. The second $\sigma(z)_j<1$ part is ensured as described in my answer. Also I can't see how I confused $j$ and $k$, which does not matter anyway as they are just indices. $\endgroup$ – M. Winter May 6 '17 at 10:30
0
$\begingroup$

If you sum over all possible classes the result must be one. Why? In your equation The bottom term get divided is simply just the sum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.