27
$\begingroup$

Is it possible to revert the softmax function in order to obtain the original values $x_i$?

$$S_i=\frac{e^{x_i}}{\sum e^{x_i}} $$

In case of 3 input variables this problem boils down to finding $a$, $b$, $c$ given $x$, $y$ and $z$:

\begin{cases} \frac{a}{a+b+c} &= x \\ \frac{b}{a+b+c} &= y \\ \frac{c}{a+b+c} &= z \end{cases}

Is this problem solvable?

$\endgroup$

2 Answers 2

25
$\begingroup$

Note that in your three equations you must have $x+y+z=1$. The general solution to your three equations are $a=kx$, $b=ky$, and $c=kz$ where $k$ is any scalar.

So if you want to recover $x_i$ from $S_i$, you would note $\sum_i S_i = 1$ which gives the solution $x_i = \log (S_i) + c$ for all $i$, for some constant $c$.

$\endgroup$
6
  • 1
    $\begingroup$ So it’s solvable up to a constant. Thank you! $\endgroup$
    – jojeck
    May 18, 2018 at 17:39
  • 1
    $\begingroup$ Which c constant should I use? There is any way of calculating it? $\endgroup$ Feb 7, 2019 at 17:16
  • 3
    $\begingroup$ @JoelCarneiro Any $c$ will work; the solution is not unique. $\endgroup$
    – angryavian
    Feb 7, 2019 at 17:58
  • 2
    $\begingroup$ In the case anybody like me spend too much time figuring out $c$: If you know your 3 input variables have to sum to 1 then your $c = (1 - log(x \cdot y \cdot z))/3)$. $\endgroup$ Sep 14, 2020 at 13:04
  • 1
    $\begingroup$ Usually they normalize the xi to sum to zero, in which case the inverse softmax would be log(x)-mean(log(x)). Or they code the first xi to be zero, in which case the inverse softmax would be log(x)-log(x[[1]]). $\endgroup$ Aug 24, 2022 at 20:12
16
$\begingroup$

The softmax function is defined as:

$$S_i = \frac{\exp(x_i)}{\sum_{j} \exp(x_j)}$$

Taking the natural logarithm of both sides:

$$\ln(S_i) = x_i - \ln(\sum_{j} \exp(x_j))$$

Rearranging the equation:

$$x_i = \ln(S_i) + \ln(\sum_{j} \exp(x_j))$$

The second term on the right-hand side is a constant over all $i$ and can be written as $C$. Therefore, we can write:

$$x_i = \ln(S_i) + C$$

This answer is adapted from this post on Reddit.

$\endgroup$
2
  • 4
    $\begingroup$ better use j in the summation $\endgroup$ Jan 22, 2021 at 18:26
  • 1
    $\begingroup$ Not quite right. There is no subscript i on the C because C is a sum over all is. Thanks for laying this out! $\endgroup$ Mar 27, 2021 at 13:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .