0
$\begingroup$

I learned that KL divergence between two Gaussian Mixtures is intractable, not easy to solve. But I am wondering if we can solve it by thinking conditional cases?

Let's say, a single multivariate Gaussian and a 2-mixture multivariate Gaussian as shown below. $$ g_{A}=Gauss(\mu_A,\sigma_A ) $$ $$ g_{B+C}=w_B\cdot Gauss(\mu_B,\sigma_B )+w_C\cdot Gauss(\mu_C,\sigma_C ) $$ Is KL divergence below correct?

$$ D_{KL} \left( g_{A}\right| \left| g_{B+C} \right)=w_B\cdot D_{KL} \left( g_{A}\right| \left| g_{B} \right)+w_C\cdot D_{KL} \left( g_{A}\right| \left| g_{C} \right) $$

seeing weights $w_B$ ,$w_C$ as conditional probabilities, $P(D_{KL}|g_B)$,$P(D_{KL}|g_C)$.

$\endgroup$

1 Answer 1

1
$\begingroup$

No, the KL divergence $D(p||q)$ is convex in the pair $(p,q)$ so the equation you wrote holds as an inequality: $$ D_{KL} \left( g_{A}\right| \left| g_{B+C} \right)\leq w_B\cdot D_{KL} \left( g_{A}\right| \left| g_{B} \right)+w_C\cdot D_{KL} \left( g_{A}\right| \left| g_{C} \right) $$ See this paper by Hershey and Olsen and also this post on related material for approximating the KL between mixtures of distributions.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .