Normalisation using Softmax- What advantage does exponential provide I am trying to apply some bench marking across different organizations. I have 3 organizations with 3 scores using which I would like to relatively rank them.
For e.g. Org 1 = 115, Org 2 = 105, Org 3= 50, then $\mathbf{x} = (115, 105, 50)$
I was told to try Softmax function
\begin{equation}
\mathrm{softmax}(\mathbf{x})=\frac{e^{x_{i}}}{\sum_{j=1}^{3}e^{x_{j}}}
\end{equation}
as it normalizes the values. I could also normalize using
\begin{equation}
\mathrm{standard~normalisation}(\mathbf{x})=\frac{x_{i}}{\sum_{j=1}^{3}x_{j}}
\end{equation}
Can anyone tell me what advantage does the Softmax function provide above the standard normalization discussed above? Does the exponential in softmax help in any specific way to increase/reduce the margin between the compared entities?
 A: Let $D=\{x_i\}$ be a dataset.
Standard normalization is generally by subtracting out the mean:
$$ \mu = \frac{1}{n}\sum_{i=1}^n x_i $$
and dividing out the standard deviation:
$$\sigma = \frac{1}{n-1}\sum_{i=1}^n (x_i-\mu)^2 $$
So that the new dataset is given by:
$$ \tilde{x}_i = \frac{x_i - \mu}{\sigma} $$
This is nice, because if you assume $x_i\sim\mathcal{N}(\mu,\sigma^2)$, then $\tilde{x}_i\sim\mathcal{N}(0,1)$.
Another, unrelated normalization, is given by:
$$ y_i = \frac{x_i - \min_j x_j}{\max_j x_j - \min_j x_j} $$
which is nice because it forces everything between zero and one.
Then there is softmax: $$ s_i = \exp(x_i)\left[ \sum_{j=1}^n \exp(x_i) \right]^{-1} $$
Here is one nice property:
$$
\sum_{i=1}^ns_i = 1
$$
which means the original dataset vector can now be interpreted as a probability distribution. (This is used in the softmax function appears in logistic regression.)
Clearly we also always have $s_i>0$. The fact that the output is always positive (regardless of whether $x_i$ is positive or negative) can be useful. 
Finally, using exponentiation can give greater separation than a linear transformation as well.
A: Below two reasons prevents your normalization output to be used in lieu of probabilities which must be restricted in $[0, 1]$. 


*

*Standard normalization as you have defined may cause negative numbers if all or some $x_i$ are negative.

*This normalization is undefined if $\sum x_i = 0$.
On the other hand using softmax as normalization isn't perfect. For example, if $x_i$ were very small but spread over large range, for example, $1 * 10^{-7}$ and $5 * 10^{-7}$ your method will produce better output while softmax would be "blind" to big relative differences. Same goes for large numbers.
The moral of the story is that softmax is not a good method for normalization, it is a good method to select maximum, especially if you want your method to be differentiable. The major advantage of softmax comes from combining it with cross entropy which has several theoretical implications such as minimizing information entropy while having computationally efficient differentiable operation. 
