# Expected Entropy Based on Dirichlet Distribution

The Dirichlet Distribution basically defines the probability that a sample came from a particular multinomial distribution if we assume that the prior probability of all multinomial distributions having generated the sample are equal.

Each multinomial distribution has a corresponding categorical distribution, and the entropy of that categorical distribution is given by

$$-\sum_x^{states}\Pr(x)\ln(\Pr(x))$$

Given a point $p=(p_1,p_2,p_3...p_n)$ randomly chosen according to a Dirichlet Distribution with parameters $k_1...k_n$, such that $\sum_ip_i=1$, the entropy of the corresponding categorical distribution is:

$$H(p)=-\sum_i^n p_i \ln(p_i)$$

What the expected value of $\text H(p)$?

In the special case where the Dirichlet Distribution is just defined by $k_1$ and $k_2$ and $p$ is 2-dimensional, the expected entropy $\text{H}(p)$ is given by the formula $$\frac{(k_1+k_2) H_{(k_1+k_2-1)}-k_1 H_{k_1}-k_2 H_{(k_2-1)}}{k_1+k_2}$$

Where $H_n$ is the $n$th harmonic number, however I haven't been able to calculate the answer for greater numbers of dimensions.

• en.wikipedia.org/wiki/Dirichlet_distribution#Entropy Is it what you want? – BGM May 5 '17 at 2:24
• No - it's not... I don't even think that equation is right, as it's giving me negative values for the entropy – J. Antonio Perez May 5 '17 at 2:52
• Did you perhaps just forget to put back in the minus sign that is in the usual (positive) convention for the entropy? – Ian May 8 '17 at 16:12
• No. I can construct an integral for the entropy I need and evaluate it numerically by calculating the entropy of random points picked according to the Dirichlet distribution, and the numeric results I get don't match the Wikipedia formula – J. Antonio Perez May 8 '17 at 16:30
• @JorgePerez The negative sign is correct. The Shannon entropy of continuous distributions can be negative. For an explanation see for example §4.b, p. 201, here: bayes.wustl.edu/etj/articles/brandeis.pdf or the book on information theory by Cover & Thomas. – pglpm Oct 5 '18 at 11:18

In a special case, there's a way to rewrite this.

In Entropy and Inference, Revisited by Nemenman et al, they note that: $$\mathbb{E}[\mathbb{H}[P]] = \psi(N_c\beta+1) - \psi(\beta+1)$$ where $N_c$ is the number of categories, $\psi$ is the digamma function, and $P\sim\text{Dir}(\beta)$ where $\beta:= \alpha_i$ parametrizes a Dirichlet with all parameters being equal (so not as general as your elegant formula).

Recalling that $$H_n = \gamma + \psi(n+1)$$ where $\gamma$ is the Euler-Mascheroni constant, we can reduce your formula as: \begin{align*} H_A - \frac{1}{A}\sum_i \alpha_i H_{\alpha_i} &= H_{N_c\beta} - \frac{1}{N_c\beta}\sum_i \beta H_\beta \\ &= \gamma + \psi(N_c\beta + 1) - \frac{1}{N_c}\sum_i H_\beta \\ &= \gamma + \psi(N_c\beta + 1) - (\gamma + \psi(\beta+1)) \\[2.1mm] &= \psi(N_c\beta + 1) - \psi(\beta+1) \\[2mm] &= \mathbb{E}[\mathbb{H}[P]] \end{align*}

So as it turns out, the general closed-form solution is

$$\text{Exp}(H(P))=H_A-\frac{1}{A}\sum _{i=1}^m \alpha _i H_{\alpha _i}$$

Where $m$ is the number of different states, $H(P)$ is the entropy of probability distribution $P$ where each state $s_i$ occors with probability $p_i$, the $\alpha _i$ are the distribution parameters of the dirichlet distribution $P$ is drawn from, and $A=\sum_{i=1}^m \alpha _i$. Each $\alpha_i = k_i+1$.

I believe the derivation is obvious.

JK the derivation isn't obvious. I found a closed-form solution when $n=2$, when $n=3$, and when $n=4$, and then I fiddled with those until I came up with a general formula that looks elegant (see above), and then tested the formula against a monte-carlo estimation of the expected value. It works but I can't prove it.

Here is a formal proof for general Dirichlet distributions $$(\alpha_1, \dots, \alpha_m)$$. I use capital $$P_i$$ to indicate that we are working with random variables.

$$-E(\sum_i P_i \log P_i)=-\sum_i E(P_i \log P_i)$$ then $$P_i \sim Beta (\alpha_i, A -\alpha_i)$$ and working with the normalizing constant you can write

$$-E_{\alpha_i, A-\alpha_i}(P_i \log P_i)= \frac{\alpha_i}{A}E_{\alpha_i+1, A-\alpha_i}(\log P_i) = \frac{\alpha_i}{A} [\psi_0 (A+1)-\psi_0(\alpha_i+1)]$$

where the last step arises by a known result (see Wikipedia page on Beta distributions): if $$X \sim Beta(\alpha, \beta)$$ then $$-E(\log P_i)= \psi_0(\alpha+\beta)-\psi_0(\alpha)$$.

Summing over $$i$$ provides your general formula.