1
$\begingroup$

I was reading this blogpost which talks about the properties of binary entropy function.

It was stated that $(1-p q) h\left(\frac{p-p q}{1-p q}\right)=h(p)+p h(q)-h(p q) \quad 0 \leq p \leq 1,0 \leq q \leq 1, p q<1$

I am unable to find this identity elsewhere. I am interested in understanding the proof of this identity. Can someone provide a reference proving this? If not, can someone provide the proof for it?

Also, could someone provide an intuition or meaningful interpretation for this property?

$\endgroup$
2
  • $\begingroup$ Surely you can prove it just using the rule $\log_2(xy)=\log_2x+\log_2y$. What would be more interesting is an interpretation of the identity. $\endgroup$
    – anon
    Oct 15, 2021 at 0:56
  • $\begingroup$ @runway44 Thanks! I will edit and add that too as a subquestion. $\endgroup$
    – wanderer
    Oct 15, 2021 at 1:01

1 Answer 1

1
$\begingroup$

Here's an interpretation.

Recall this property: the entropy of a non overlapping mixture of distributions equals the mixture (weighed average) of the individual entropies, plus the entropy of the mixing factor (binary entropy).

Now, consider a random variable $X$ taking $3$ values with probabilities $${\bf p} = [a; b;c] \tag 1$$

with $a+b+c=1$.

This can be considered as a mixture of two non-overlapping distributions thus: $$ a [1; 0; 0] + (1-a) \left[ 0; \frac{b}{1-a} ; \frac{c}{1-a}\right] \tag 2 $$

Hence $$H(X) = a 0 + (1-a) h\left(\frac{b}{1-a} \right) + h(a) = (1-a) h\left(\frac{b}{1-a} \right) + h(a) \tag3 $$

But $(1)$ it can also be decomposed in this way:

$$ b [0; 1; 0] + (1-b) \left[ \frac{a}{1-b} ; 0; \frac{c}{1-b}\right] \tag 4 $$

Hence $$H(X) = (1-b) h\left(\frac{a}{1-b} \right) + h(b) \tag5 $$

or

$$ (1-b) h\left(\frac{a}{1-b} \right) + h(b) = (1-a) h\left(\frac{b}{1-a} \right) + h(a) \tag 6$$

Letting $1-a=p$ and $ b = pq$ we get the original equation.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .