4
$\begingroup$

The Renyi entropy assigns a real number $H_{\vec{p}} (\alpha)$ to a probability distribution $\vec{p} = (p_1, p_2, \ldots)$ as a function of an order parameter $\alpha > 0, \alpha \neq 1$:

$$H_{\vec{p}} (\alpha) \equiv \frac{1}{1-\alpha}\log\Bigg(\sum_i p_i^\alpha\Bigg).$$

When considered as a function of $\alpha$, this has been called the "spectrum of Renyi information". The Renyi entropies are notable for being non-increasing under doubly stochastic operations ($\vec{p} \to T\vec{p}$ for doubly stochastic matrix $T$) and for recovering the Shannon entropy as $\alpha \to 1$.

Let us ignore the order of the $p_i$, e.g., by focusing on the set of distributions where the probabilities are in non-increasing order. If we take the Renyi entropy to map such distributions to functions of $\alpha$, $$ \vec{p} \to H_{\vec{p}} (\alpha),$$ is this map invertible? If so, does the inverse have a closed form?

In other words, must two discrete probability distributions be permutations of each other if they have the same Renyi entropy at all orders? Commenter Mike Earnest suspects so. And indeed, since large values of $\alpha$ "pick out" only the very largest probabilities, while smaller values are progressively more sensitive to further down the distribution, it seems likely to be true. This is essentially asking whether probability distributions are uniquely determined by their $\alpha$-norm $\|p_i\|_\alpha \equiv \left(\sum_i p_i^\alpha\right)^{1/\alpha}$ to all orders.

Bonus question: if we generalize to continuous probability distributions, will the same be true up to a set of measure zero?

$\endgroup$
3
$\begingroup$

The answer to this question is "yes" in the case of discrete probabilities. I will gesture toward the solution with a physics-y argument but not give anything rigorous. Anyone who wants to contribute something better is strongly encouraged to do so.

Let $f(\alpha) \equiv \sum_i p_i^\alpha = \|p_i\|_\alpha^\alpha = \exp\left[(1-\alpha)H_{\vec p}(\alpha)\right]$ for $\alpha > 0, \neq 1$. Analytically continuing to the complex plane, this looks like a discrete Fourier transform of a "support" function on the probabilities: $f(it) = \sum_i e^{i \ln(p_i) t}$. We can apply the inverse Fourier transform to get that support function:

$$\tilde{f}(\omega) \equiv \int_{-\infty}^\infty \! dt \, f(it) e^{-i \omega t} = \sum_i \int_{-\infty}^\infty \! dt \,e^{i(\ln(p_i) -\omega) t} = \sum_i \delta(\ln(p_i)-\omega)$$ where $\delta$ represents the Diract delta function (distribution).

Therefore, it ought to be possible to read off the probabilities, including the degree of any degeneracies, by computing $\tilde{f}(\omega)$ and finding the peaks. Given the nature of the discrete Fourier transform and analytic continuation, it's presumably only necessary to know $f(\alpha)$ for a countable set.

$\endgroup$
  • $\begingroup$ Source: I heard this from Dan Ranard. $\endgroup$ – Jess Riedel Jan 10 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.