5
$\begingroup$

Look at the Wiki page for Softmax function (section "Smooth approximation of maximum"): https://en.wikipedia.org/wiki/Softmax_function

It is saying that the following is a smooth approximation to the softmax: $$ \mathcal{S}_{\alpha}\left(\left\{x_i\right\}_{i=1}^{n}\right) = \frac{\sum_{i=1}^{n}x_i e^{\alpha x_i}}{\sum_{i=1}^{n}e^{\alpha x_i}} $$

  • Is it an approximation to the Softmax?

    • If so, Softmax is already smooth; why do we create another smooth approximation?

    • If so, how do derive it from Softmax?

  • I don't see why this might be better than Softmax for gradien descent updates.

$\endgroup$
2
  • 1
    $\begingroup$ I am confused by this as well, but let me make a comment that might be useful. A smooth approximation of maximum that I am familiar with is $f(x,\alpha):=\alpha^{-1} \log\left(\sum_i e^{\alpha x_i}\right)$ which is always within an additive $(\log n)/\alpha$ from the maximum. The function in your question is $\partial f/\partial\alpha$. $\endgroup$
    – Sasho Nikolov
    May 18, 2015 at 4:46
  • $\begingroup$ Out of curiosity, what problem are you solving? Are you sure you can't just use the max function? Many convex optimization algorithms can handle nondifferentiable objective functions. $\endgroup$
    – littleO
    Jul 18, 2015 at 5:13

1 Answer 1

Reset to default
3
$\begingroup$

This is a smooth approximation of maximum function:

$$ \max\{x_1,\dots, x_n\} $$

where $\alpha$ controls the "softness" of the maximum. The detailed explanation is available here: http://www.johndcook.com/blog/2010/01/13/soft-maximum/

Softmax is better then maximum, because it is smooth function, while $\max$ is not smooth and does not always have a gradient.

$\endgroup$
8
  • $\begingroup$ Could you perhaps sum up the detailed discussion in your answer? $\endgroup$
    – Ali Caglayan
    Sep 24, 2014 at 22:21
  • $\begingroup$ The sum up is the first sentence of my reply, in the explanation there are also visual examples given. $\endgroup$
    – coffee
    Sep 25, 2014 at 23:26
  • 1
    $\begingroup$ to be short, $\alpha \to +\infty$ makes softmax converge to max, and $\alpha \to -\infty$ makes softmax converge to min $\endgroup$
    – coffee
    Sep 26, 2014 at 2:53
  • $\begingroup$ The blog entry discusses a different function from the one in the question. $\endgroup$
    – Sasho Nikolov
    May 18, 2015 at 4:42
  • $\begingroup$ yes, the function described there is a little different (it is numerically better I think, this is the one you've mentioned), but the idea is the same. Both are valid approximations. $\endgroup$
    – coffee
    May 21, 2015 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.