Understanding the Softmax Function I'm trying the understand how multiplying scores affect the softmax function.
$\text{Softmax}(X_i)  = \frac{e^{X_i}}{\Sigma(e^{X_i)}}$
So I know that when scores ($X_i$) are multiplied by 10, the resulting softmax probabilities are closer to 1 or closer to 0 than before.
Conversely, So I know that when scores ($X_i$) are divided by 10, the resulting softmax probabilities are closer to the uniform distribution than before.
I got this answer intuitively (more extreme scores $\rightarrow$ more extreme probabilities). But how can I arrive to this conclusion mathematically?
(I'm a Statistics student, apologies for my ignorance in analysis )
 A: Let $S:{\mathbb R^n} \longrightarrow {\mathbb R^n}$ be the softmax function given by $$S(X_i) := \frac{\exp{X_i}}{\sum_{j=1}^n \exp{X_j}}$$ as you've said.  We can see immediately that $0\leq S(X_i) \leq 1$ always, so it's reasonable to consider the values of $S(X_i)$ as probabilities.
When we multiply the scores by a constant as you describe (by $10$ or $0.1$ in your examples) we are multiplying each vector entry by the same scalar.  It's slightly more general to consider the dot product with a weight-vector though, so we'll write $a\cdot X$, where $a=(10,10,...10)$ yields your first example.
So, observing that $(a\cdot X)_i$ = $a_iX_i$ we have:
$$
\begin{eqnarray}
S(a_i \cdot X_i) & = & \frac{e^{a_i X_i}}{\sum_{j=1}^n e^{a_j X_j} } \\
{} & = & \frac{{(e^{X_i}})^{a_i}}{\sum_{j=1}^n {(e^{X_j}})^{a_j} }
\end{eqnarray}
$$
We can understand the behaviour of the softmax function by understanding what happens as $a_i \rightarrow \infty$.  We can rewrite $S(X_i)$ simply by dividing the top and bottom of the fraction by the top:
$$
\begin{eqnarray}
S(a_i \cdot X_i) & = & \left({\sum_{j=1}^n \frac{{(e^{X_j}})^{a_j}}{{(e^{X_i}})^{a_i}}} \right)^{-1} \\
S(a_i \cdot X_i) & = & \left({\sum_{j=1}^n \left(\frac{e^{X_j}}{e^{X_i}} \right)^{a_i}} \right)^{-1} 
\end{eqnarray}
$$
Now, when $j=i$ in this fraction we get $1$ appearing.  Otherwise we get numbers that are smaller than $1$ (if $X_i > X-j$) or greater than $1$ (if $X_i < X_j$).  As $a_i \rightarrow\infty$ the numbers greater than $1$ head towards $\infty$ and those less than $1$ head towards $0$.  If any number heads towards $\infty$ the whole fraction heads to $0$.
So, since the only time all numbers will head towards $0$ is when we divide by $X_{max} := \max_i X_i$, we see that $S(a_i \cdot X_i) \rightarrow 0$ for $X_i \not= X_{max}$ and $S(a_i \cdot X_{max}) \rightarrow 1$.  At this point $S$ is selecting the maximum value, which is where the "softmax" name comes from.
Finally note that this breaks if there are two (or more) largest values of $X_i$ (e.g. $X=(1,2,5,10,10,10)^T$).  However, in this case there is no single maximum either, and this indicates that you should use a different approach.
A: I know it is an old question but IMHO it is more clear to see it this way.
Again, the softmax function $ S $ can be considered a Vector Field $ S:R^n \rightarrow R^n $ where input $ X $ has $ n $ elements and the output $ S(X) $ has also $ n $ elements. So the element $ i $ of $ S(X) $ which will be denoted by $ S(X)_i $ it is defined by:
$$
S(X)_i:= \frac{e^{X_i}}{\sum_{j=0}^n e^{Xj}} \Rightarrow S(aX)_i:= \frac{e^{aX_i}}{\sum_{j=0}^n e^{aXj}}
$$
So, the annoying part is the denominator. Let's call $ X_{(1)} $ and $ X_{(2)} $ the first two order statistics of the $ n $ elements of X (the maximum and the next to the maximum), those are scalars not vectors. In general:
$$
X_{(1)} \geq X_{(2)} \geq ... \geq X_{(n)} \Rightarrow S(X)_{(1)} = \frac{e^{X_{(1)}}}{\sum_{j=0}^n e^{Xj}} \geq S(X)_{(2)} \geq ... \geq S(X)_{(n)}
$$
With this, you can see that:
$$
\frac{S(X)_{(1)}}{S(X)_{(2)}} = e^{X_{(1)} - X_{(2)}} \geq 1
$$
So, with the fraction we get rid off the denominator, the key is that:
$$
\frac{S(aX)_{(1)}}{S(aX)_{(2)}} = e^{aX_{(1)} - aX_{(2)}} = (e^{X_{(1)} - X_{(2)}})^a = (\frac{S(X)_{(1)}}{S(X)_{(2)}})^a 
$$
So now, if $ a \gt 1$ then:
$$ 
\frac{S(aX)_{(1)}}{S(aX)_{(2)}} \geq \frac{S(X)_{(1)}}{S(X)_{(2)}} \
$$
which means that the distribution will be more "spiky", the diferences between the order statistics will get bigger. In the limit $ a \rightarrow  \infty $ all the "probability mass" will concentrate on $ S(aX)_{(1)} \Rightarrow S(aX)_{(1)} \rightarrow 1$.
On the other hand if $ a \rightarrow 0 $ then:
$$
\frac{S(aX)_{(1)}}{S(aX)_{(2)}} = (\frac{S(X)_{(1)}}{S(X)_{(2)}})^a \rightarrow 1
$$
which means that in the limit all the values will be equal $ S(aX)_{i} \rightarrow \frac{1}{n} $, the smaller $ a $ the more uniform the distribution will be, the differences will get smaller.
In a lot of contexts instead of the $a$ parameter multiplying $X$ like the used in the explanation above, the function is parametrized with $a = \frac{1}{t}$ where $t$ is called temperature, i.e when the temperature grows the distribution "melts" into a uniform distribution:
$$
S(\frac{X}{t})_i:= \frac{e^{\frac{X_i}{t}}}{\sum_{j=0}^n e^{\frac{X_j}{t}}}
$$
