I learned that KL divergence between two Gaussian Mixtures is intractable, not easy to solve. But I am wondering if we can solve it by thinking conditional cases?

Let's say, a single multivariate Gaussian and a 2-mixture multivariate Gaussian as shown below. $$ g_{A}=Gauss(\mu_A,\sigma_A ) $$ $$ g_{B+C}=w_B\cdot Gauss(\mu_B,\sigma_B )+w_C\cdot Gauss(\mu_C,\sigma_C ) $$ Is KL divergence below correct?

$$ D_{KL} \left( g_{A}\right| \left| g_{B+C} \right)=w_B\cdot D_{KL} \left( g_{A}\right| \left| g_{B} \right)+w_C\cdot D_{KL} \left( g_{A}\right| \left| g_{C} \right) $$

seeing weights $w_B$ ,$w_C$ as conditional probabilities, $P(D_{KL}|g_B)$,$P(D_{KL}|g_C)$.


1 Answer 1


No, the KL divergence $D(p||q)$ is convex in the pair $(p,q)$ so the equation you wrote holds as an inequality: $$ D_{KL} \left( g_{A}\right| \left| g_{B+C} \right)\leq w_B\cdot D_{KL} \left( g_{A}\right| \left| g_{B} \right)+w_C\cdot D_{KL} \left( g_{A}\right| \left| g_{C} \right) $$ See this paper by Hershey and Olsen and also this post on related material for approximating the KL between mixtures of distributions.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .