2
$\begingroup$

A random variable $X$ takes positive integer values and $E[X]=6$. What distribution of the random variable $X$ maximises the entropy $H(X)$? What if $X$ can only take a finite number of values?

Ignoring the expected value bit, effectively I want to take some form of uniform distribution on the natural numbers (I know this can't be done) since if $X$ takes a finite number of values then the distribution which maximises entropy is the uniform distribution on those values.

I have a feeling that you might not be able to maximise the entropy in the infinite value case but I don't know how I would show that.

Any help or hints appreciated.

$\endgroup$

1 Answer 1

2
$\begingroup$

The following heuristic argument suggests that the entropy $H(X)$ is largest when $$p_k={1\over5}\left({5\over6}\right)^k\qquad(k\geq1)\ .\tag{1}$$ The value obtained in this way is $$H(X)=6\log 6-5\log 5\ .$$ We have to maximize $$H({\bf p}):=-\sum\nolimits_{k\geq1} p_k\log p_k$$ under the constraints $$\sum\nolimits_{k\geq1} p_k=1,\quad \sum\nolimits_{k\geq1}k p_k=6\ .\tag{2}$$ We set up the "Lagrangian" $$\Phi:=-\sum\nolimits_{k\geq1} p_k\log p_k-\lambda\left(\sum\nolimits_{k\geq1} p_k-1\right)-\mu\left(\sum\nolimits_{k\geq1}k p_k-6\right)$$ and obtain the equations $$\Phi_{.k}=-\log p_k -1-\lambda -\mu k=0\qquad(k\geq1)\ .$$ It follows that $$p_k=Ae^{-Bk}\qquad(k\geq1)\ ,$$ where $A$ and $B$ have to be determined using the constraints $(2)$. Doing the computation one finds the $p_k$ given in $(1)$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .