1
$\begingroup$

I know for continuous distribution, given mean and variance, it's Normal distribution.

I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance.

I assumed differential entropy would be a intelligent guess, but it was lower than some empirical entropy with the same variance.

So I wonder how I can determine the maximum entropy for a discrete distribution when the mean and the variance is fixed...

for example, let's say the support is 1~100 all integers, and the mean is 61.09 and the sd is 12.89. what would be the maximum entropy? I tried the differential entropy 1/2*(1+ln(2*pi*sd^2)) = 3.97 but I found my empirical distribution has the entropy of 5.49

28 29 33 35 36 37 38 39 41 42 43 44 0.005 0.005 0.005 0.005 0.005 0.015 0.010 0.005 0.005 0.005 0.015 0.020 45 46 47 48 49 50 51 52 53 54 55 56 0.020 0.010 0.020 0.030 0.025 0.010 0.020 0.025 0.015 0.025 0.025 0.015 57 58 59 60 61 62 63 64 65 66 68 69 0.065 0.020 0.020 0.030 0.055 0.035 0.040 0.005 0.030 0.015 0.025 0.050 70 71 72 73 74 75 76 77 78 79 80 81 0.030 0.030 0.025 0.020 0.020 0.015 0.015 0.005 0.005 0.020 0.010 0.005 82 83 85 86 87 88 89 92 0.015 0.015 0.010 0.005 0.005 0.010 0.005 0.005

Related question:

https://math.stackexchange.com/questions/957680/maximum-entropy-distribution-with-given-variance

$\endgroup$
2
$\begingroup$

Disclaimer: I haven't worked out the math for this particular case.

I'd recommend reading chapter 12 of Cover & Thomas' Elements of Information Theory 2e.

Basically, to find it, you'd use Lagrange multipliers/KKT conditions (just like in the continuous case). If the multipliers exist, you have your distribution.

For what its worth, the mean constraint only gives something geometric-y.

For relating differential entropy and entropy of quantized distributions (which seems to be something you tried to do), see ch. 8 of Cover & Thomas.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.