# Proving an identity involving binary entropy

I was reading this blogpost which talks about the properties of binary entropy function.

It was stated that $$(1-p q) h\left(\frac{p-p q}{1-p q}\right)=h(p)+p h(q)-h(p q) \quad 0 \leq p \leq 1,0 \leq q \leq 1, p q<1$$

I am unable to find this identity elsewhere. I am interested in understanding the proof of this identity. Can someone provide a reference proving this? If not, can someone provide the proof for it?

Also, could someone provide an intuition or meaningful interpretation for this property?

• Surely you can prove it just using the rule $\log_2(xy)=\log_2x+\log_2y$. What would be more interesting is an interpretation of the identity.
– anon
Oct 15, 2021 at 0:56
• @runway44 Thanks! I will edit and add that too as a subquestion. Oct 15, 2021 at 1:01

Here's an interpretation.

Recall this property: the entropy of a non overlapping mixture of distributions equals the mixture (weighed average) of the individual entropies, plus the entropy of the mixing factor (binary entropy).

Now, consider a random variable $$X$$ taking $$3$$ values with probabilities $${\bf p} = [a; b;c] \tag 1$$

with $$a+b+c=1$$.

This can be considered as a mixture of two non-overlapping distributions thus: $$a [1; 0; 0] + (1-a) \left[ 0; \frac{b}{1-a} ; \frac{c}{1-a}\right] \tag 2$$

Hence $$H(X) = a 0 + (1-a) h\left(\frac{b}{1-a} \right) + h(a) = (1-a) h\left(\frac{b}{1-a} \right) + h(a) \tag3$$

But $$(1)$$ it can also be decomposed in this way:

$$b [0; 1; 0] + (1-b) \left[ \frac{a}{1-b} ; 0; \frac{c}{1-b}\right] \tag 4$$

Hence $$H(X) = (1-b) h\left(\frac{a}{1-b} \right) + h(b) \tag5$$

or

$$(1-b) h\left(\frac{a}{1-b} \right) + h(b) = (1-a) h\left(\frac{b}{1-a} \right) + h(a) \tag 6$$

Letting $$1-a=p$$ and $$b = pq$$ we get the original equation.