1
$\begingroup$

Background: The gumbel-softmax distribution has been introduced in 1 and 2 for its useful application in neural networks, due to the ability to use backprop in gradient-based approximate bayesian inference. The distribution is

$$f_{\text{gumbel-softmax-k}}(y_1, \cdots, y_k;\pi, \tau) = \Gamma(k)\tau^{k-1}\bigg(\sum_{i=1}^k\pi_i/y_i^\tau\bigg)^{-k}\prod_{i=1}^k(\pi_i /y_i^{\tau + 1})$$

where the $\pi_i$ represent $k$ discrete class probabilities such that $\pi_i \ge 0$ and $\sum_i\pi_i=1$, with samples on the same domain $y_i \ge 0$ and $\sum_i y_i=1$. You may notice as the temperature $\tau \rightarrow 0$, the discrete categorical softmax distribution is recovered. It is a continuous relaxation of the discrete Categorical distribution of $k$ dimensions: $ p(y) = \pi_1^{\delta_{y,1}}\cdots\pi_k^{\delta_{y,k}}$ which comes from the gumbel max trick, where we can sample categorical variables from class probabilities $\pi_1, \cdots, \pi_k$ using $k \sim \text{argmax}_i[\log \pi_i + g_i]$ where $g_i \sim \text{Gumbel}(0, 1)$. Please see references 1 and 2 linked above for more details on this distribution.

Question 1: What is the entropy of this gumbel-softmax distribution for arbitrary $k$?

$$ \mathcal{H}_{\text{gumbel-softmax-k}}(\pi_1, \cdots, \pi_k, \tau) = -\int_{\bf{y}}f({\bf y};\pi_1, \cdots, \pi_k, \tau)\log f({\bf y};\pi_1, \cdots, \pi_k, \tau)d{\bf y} $$

Question 2: In particular I am interested in the case where case where $k=2$ with the parameterization $y_1 = y$, and $y_2 = 1 - y\ $ (and $\pi = \pi_1$, and $\pi_2 = 1 - \pi$) which is a continuous relaxation on the $Bernoulli$ distribution:

$$f_{\text{gumbel-softmax-2}}(y;\pi, \tau) = \Gamma(2)\ \tau \ \bigg(\frac{\pi}{y^{\tau}} + \frac{1-\pi}{(1-y)^{\tau}}\bigg)^{-2}\frac{(\pi-\pi^2)}{(y - y^2)^{\tau+1}}dy$$

$$\mathcal{H}_{\text{gumbel-softmax-2}}(\pi, \tau) = - \int_{y=0}^1f(y;\pi, \tau)\log f(y;\pi,\tau)dy$$

Edit: Bump

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.