24
$\begingroup$

Is it possible to revert the softmax function in order to obtain the original values $x_i$?

$$S_i=\frac{e^{x_i}}{\sum e^{x_i}} $$

In case of 3 input variables this problem boils down to finding $a$, $b$, $c$ given $x$, $y$ and $z$:

\begin{cases} \frac{a}{a+b+c} &= x \\ \frac{b}{a+b+c} &= y \\ \frac{c}{a+b+c} &= z \end{cases}

Is this problem solvable?

$\endgroup$

2 Answers 2

21
$\begingroup$

Note that in your three equations you must have $x+y+z=1$. The general solution to your three equations are $a=kx$, $b=ky$, and $c=kz$ where $k$ is any scalar.

So if you want to recover $x_i$ from $S_i$, you would note $\sum_i S_i = 1$ which gives the solution $x_i = \log (S_i) + c$ for all $i$, for some constant $c$.

$\endgroup$
5
  • 1
    $\begingroup$ So it’s solvable up to a constant. Thank you! $\endgroup$
    – jojek
    May 18, 2018 at 17:39
  • 1
    $\begingroup$ Which c constant should I use? There is any way of calculating it? $\endgroup$ Feb 7, 2019 at 17:16
  • 3
    $\begingroup$ @JoelCarneiro Any $c$ will work; the solution is not unique. $\endgroup$
    – angryavian
    Feb 7, 2019 at 17:58
  • $\begingroup$ Any $c$ will work, one choice is if you augment the $x_i$ vector like $(0, x_1,...,x_n)$ then this will induce a particular $c$, note the corresponding log-sum-exp -- the gradient of which is the softmax -- would also be convex (en.wikipedia.org/wiki/LogSumExp). $\endgroup$ Jul 29, 2019 at 10:59
  • $\begingroup$ In the case anybody like me spend too much time figuring out $c$: If you know your 3 input variables have to sum to 1 then your $c = (1 - log(x \cdot y \cdot z))/3)$. $\endgroup$ Sep 14, 2020 at 13:04
13
$\begingroup$

A similar question was asked in a post of reddit. The answer below is adapted from that post:

$S_{i}$ = $\exp(x_{i})/(\sum_{i} \exp(x_{i}))$

Taking ln on both sides:

$\ln(S_{i}) = x_{i} - \ln(\sum_{i} \exp(x_{i}))$

Changing sides:

$x_{i} = \ln(S_{i}) + \ln(\sum_{i} \exp(x_{i}))$

The second term of the right hand side is constant for a particular $i$ and can be written as $C_{i}$. Therefore, we can write:

$x_{i} = \ln(S_{i}) + C$

$\endgroup$
2
  • 1
    $\begingroup$ better use j in the summation $\endgroup$ Jan 22, 2021 at 18:26
  • 1
    $\begingroup$ Not quite right. There is no subscript i on the C because C is a sum over all is. Thanks for laying this out! $\endgroup$ Mar 27, 2021 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.