Tag Info

New answers tagged


Usually one requires certain properties (axioms) for an entropy: Nonnegativity; the bigger the uncertainty, the bigger the entropy; and additivity for independent observations/measurements. The last property implies that there should be a logarithmic dependence. For more details on the axiomatic formulation of entropy see ...


You are simply looking for the solutions of the implicit equation: $x \log (x)+y\log (y)+(1-x-y)\log (1-x-y) = 0.2\log (0.2)+0.3\log (0.3)+ 0.5 \log (0.5)$ For example, you can try a plot here


There are such pmf coincidences, applicable for any set of candidates with more than 2 candidate values other than the maximal entropy all-value-equal entropy. For your example, consider (although there is an entire 1-parameter family of iso-entropic distributions) the distrbution with $$ p(X) = [0.24301892,0.24301892,0.51396216] $$ This has the identical ...


From our definition of conditional entropy: $$ H(Y|X) = \sum_{x}p_{X}(x)H(Y|X=x) = -\mathbb{E}[\text{log}p(Y|X)] $$ We know the chain rule which states that: $$ H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y) $$ so $$ H(Y|X) - H(X|Y) = H(Y) - H(X) = 0 $$ by the fact that X and Y have the same distribution. Of course this is clear from the symmetry.

Top 50 recent answers are included