17
$\begingroup$

Is it possible to revert the softmax function in order to obtain the original values $x_i$?

$$S_i=\frac{e^{x_i}}{\sum e^{x_i}} $$

In case of 3 input variables this problem boils down to finding $a$, $b$, $c$ given $x$, $y$ and $z$:

\begin{cases} \frac{a}{a+b+c} &= x \\ \frac{b}{a+b+c} &= y \\ \frac{c}{a+b+c} &= z \end{cases}

Is this problem solvable?

$\endgroup$
17
$\begingroup$

Note that in your three equations you must have $x+y+z=1$. The general solution to your three equations are $a=kx$, $b=ky$, and $c=kz$ where $k$ is any scalar.

So if you want to recover $x_i$ from $S_i$, you would note $\sum_i S_i = 1$ which gives the solution $x_i = \log (S_i) + c$ for all $i$, for some constant $c$.

$\endgroup$
5
  • 1
    $\begingroup$ So it’s solvable up to a constant. Thank you! $\endgroup$ – jojek May 18 '18 at 17:39
  • 1
    $\begingroup$ Which c constant should I use? There is any way of calculating it? $\endgroup$ – Joel Carneiro Feb 7 '19 at 17:16
  • 2
    $\begingroup$ @JoelCarneiro Any $c$ will work; the solution is not unique. $\endgroup$ – angryavian Feb 7 '19 at 17:58
  • $\begingroup$ Any $c$ will work, one choice is if you augment the $x_i$ vector like $(0, x_1,...,x_n)$ then this will induce a particular $c$, note the corresponding log-sum-exp -- the gradient of which is the softmax -- would also be convex (en.wikipedia.org/wiki/LogSumExp). $\endgroup$ – Josh Albert Jul 29 '19 at 10:59
  • $\begingroup$ In the case anybody like me spend too much time figuring out $c$: If you know your 3 input variables have to sum to 1 then your $c = (1 - log(x \cdot y \cdot z))/3)$. $\endgroup$ – Rasmus Ø. Pedersen Sep 14 '20 at 13:04
9
$\begingroup$

A similar question was asked in a post of reddit. The answer below is adapted from that post:

$S_{i}$ = $\exp(x_{i})/(\sum_{i} \exp(x_{i}))$

Taking ln on both sides:

$\ln(S_{i}) = x_{i} - \ln(\sum_{i} \exp(x_{i}))$

Changing sides:

$x_{i} = \ln(S_{i}) + \ln(\sum_{i} \exp(x_{i}))$

The second term of the right hand side is constant for a particular $i$ and can be written as $C_{i}$. Therefore, we can write:

$x_{i} = \ln(S_{i}) + C_{i}$

$\endgroup$
1
  • $\begingroup$ better use j in the summation $\endgroup$ – eyaler Jan 22 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.