2
$\begingroup$

I was looking in KL divergence lemma, but could not figure out how they derive the additive property of KL divergence. For reference KL divergence which is a measure of difference between two probability distributions $P$ and $Q$ is defined as $D_{KL}(P||Q)=\sum_i \log{\frac{P(i)}{Q(i)}}P(i)$, where $i$ is the support of $P$ and $Q$.Now as mentioned in wikipedia.

The Kullback–Leibler divergence is additive for independent distributions in much the same way as Shannon entropy. If ${P_{1},P_{2}} $ are independent distributions, with the joint distribution $P(x,y)=P_{1}(x)P_{2}(y)$ , and $Q,Q_{1},Q_{2}$ likewise, then $D_{KL}(P||Q)=D_{KL}(P_1||Q_1)+D_{KL}(P_2||Q_2)$

If I split up the $log$ in the definition I get the following $D_{KL}(P||Q)=D_{KL}(P_1||Q_1)\sum P_2 +D_{KL}(P_2||Q_2)\sum P_1$

$\endgroup$
1
$\begingroup$

Remember that if you sum up a probability mass function for all it's values, then the result has to be one (it is an axiom of the probability distributions), so you have $D_{KL}(P||Q)=D_{KL}(P_1||Q_1)\sum P_2+D_{KL}(P_2||Q_1)\sum P_1$, as I stated before: $\sum P_1=1$ and $\sum P_2=1$, so then: $D_{KL}(P||Q)=D_{KL}(P_1||Q_1)\sum P_2+D_{KL}(P_2||Q_1)\sum P_1=D_{KL}(P_1||Q_1)+D_{KL}(P_2||Q_2)$, so there you have the additive property.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.