3
$\begingroup$

The Dirichlet Distribution basically defines the probability that a sample came from a particular multinomial distribution if we assume that the prior probability of all multinomial distributions having generated the sample are equal.

Each multinomial distribution has a corresponding categorical distribution, and the entropy of that categorical distribution is given by

$$-\sum_x^{states}\Pr(x)\ln(\Pr(x))$$

Given a point $p=(p_1,p_2,p_3...p_n)$ randomly chosen according to a Dirichlet Distribution with parameters $k_1...k_n$, such that $\sum_ip_i=1$, the entropy of the corresponding categorical distribution is:

$$H(p)=-\sum_i^n p_i \ln(p_i)$$

What the expected value of $\text H(p)$?

In the special case where the Dirichlet Distribution is just defined by $k_1$ and $k_2$ and $p$ is 2-dimensional, the expected entropy $\text{H}(p)$ is given by the formula $$\frac{(k_1+k_2) H_{(k_1+k_2-1)}-k_1 H_{k_1}-k_2 H_{(k_2-1)}}{k_1+k_2}$$

Where $H_n$ is the $n$th harmonic number, however I haven't been able to calculate the answer for greater numbers of dimensions.

$\endgroup$
  • $\begingroup$ en.wikipedia.org/wiki/Dirichlet_distribution#Entropy Is it what you want? $\endgroup$ – BGM May 5 '17 at 2:24
  • $\begingroup$ No - it's not... I don't even think that equation is right, as it's giving me negative values for the entropy $\endgroup$ – J. Antonio Perez May 5 '17 at 2:52
  • $\begingroup$ Did you perhaps just forget to put back in the minus sign that is in the usual (positive) convention for the entropy? $\endgroup$ – Ian May 8 '17 at 16:12
  • $\begingroup$ No. I can construct an integral for the entropy I need and evaluate it numerically by calculating the entropy of random points picked according to the Dirichlet distribution, and the numeric results I get don't match the Wikipedia formula $\endgroup$ – J. Antonio Perez May 8 '17 at 16:30
  • $\begingroup$ @JorgePerez The negative sign is correct. The Shannon entropy of continuous distributions can be negative. For an explanation see for example §4.b, p. 201, here: bayes.wustl.edu/etj/articles/brandeis.pdf or the book on information theory by Cover & Thomas. $\endgroup$ – pglpm Oct 5 '18 at 11:18
1
$\begingroup$

In a special case, there's a way to rewrite this.

In Entropy and Inference, Revisited by Nemenman et al, they note that: $$ \mathbb{E}[\mathbb{H}[P]] = \psi(N_c\beta+1) - \psi(\beta+1) $$ where $N_c$ is the number of categories, $\psi$ is the digamma function, and $P\sim\text{Dir}(\beta)$ where $\beta:= \alpha_i$ parametrizes a Dirichlet with all parameters being equal (so not as general as your elegant formula).

Recalling that $$ H_n = \gamma + \psi(n+1) $$ where $\gamma$ is the Euler-Mascheroni constant, we can reduce your formula as: \begin{align*} H_A - \frac{1}{A}\sum_i \alpha_i H_{\alpha_i} &= H_{N_c\beta} - \frac{1}{N_c\beta}\sum_i \beta H_\beta \\ &= \gamma + \psi(N_c\beta + 1) - \frac{1}{N_c}\sum_i H_\beta \\ &= \gamma + \psi(N_c\beta + 1) - (\gamma + \psi(\beta+1)) \\[2.1mm] &= \psi(N_c\beta + 1) - \psi(\beta+1) \\[2mm] &= \mathbb{E}[\mathbb{H}[P]] \end{align*}

$\endgroup$
1
$\begingroup$

So as it turns out, the general closed-form solution is

$$\text{Exp}(H(P))=H_A-\frac{1}{A}\sum _{i=1}^m \alpha _i H_{\alpha _i}$$

Where $m$ is the number of different states, $H(P)$ is the entropy of probability distribution $P$ where each state $s_i$ occors with probability $p_i$, the $\alpha _i$ are the distribution parameters of the dirichlet distribution $P$ is drawn from, and $A=\sum_{i=1}^m \alpha _i$. Each $\alpha_i = k_i+1$.

I believe the derivation is obvious.

JK the derivation isn't obvious. I found a closed-form solution when $n=2$, when $n=3$, and when $n=4$, and then I fiddled with those until I came up with a general formula that looks elegant (see above), and then tested the formula against a monte-carlo estimation of the expected value. It works but I can't prove it.

$\endgroup$
1
$\begingroup$

Here is a formal proof for general Dirichlet distributions $(\alpha_1, \dots, \alpha_m)$. I use capital $P_i$ to indicate that we are working with random variables.

$$-E(\sum_i P_i \log P_i)=-\sum_i E(P_i \log P_i)$$ then $P_i \sim Beta (\alpha_i, A -\alpha_i)$ and working with the normalizing constant you can write

$$ -E_{\alpha_i, A-\alpha_i}(P_i \log P_i)= \frac{\alpha_i}{A}E_{\alpha_i+1, A-\alpha_i}(\log P_i) = \frac{\alpha_i}{A} [\psi_0 (A+1)-\psi_0(\alpha_i+1)] $$

where the last step arises by a known result (see Wikipedia page on Beta distributions): if $X \sim Beta(\alpha, \beta)$ then $-E(\log P_i)= \psi_0(\alpha+\beta)-\psi_0(\alpha)$.

Summing over $i$ provides your general formula.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.