Given $Z\sim$ Gaussian Mixture model: $$p(z) = \sum_{i=1}^n \pi_i \cdot g_i(z) $$ with $g_i(z) = \mathcal{N}(z;\mu_i,\sigma^2_i),$ define $\phi = \{\mu_i,\sigma^2_i,\pi_i\}_{i=1}^n.$

I want to find a low variance estimate for the entropy $\nabla_\phi \mathcal{H}(q(z)).$ At least for the gradients with respect to $\mu_i,\sigma^2_i$ I was thinking of writing:

$$ \nabla \mathcal{H}(q(z)) = \sum_{i=1}^n \pi_i \cdot \nabla \mathbb{E_{Z\sim g_i(\cdot)}[\log p(z)]} $$ together with the $\epsilon$ reparametrization trick for each component $g_i(\cdot).$

Is there a better way of doing it, i.e. estimator with lower variance?

What would be the best way to compute derivatives with respect to the mixture weights? We could just have the same approach as above. In this case, would we know if this estimator has high variance?

How could I test this empirically?


  • $\begingroup$ what is $\mathcal{H}$? $\endgroup$ – George Dewhirst Nov 27 at 3:37
  • $\begingroup$ @GeorgeDewhirst, I'm pretty sure it's the entropy $\endgroup$ – Thoth Nov 27 at 4:05
  • $\begingroup$ @GeorgeDewhirst edited. entropy indeed $\endgroup$ – Dan Leonte Nov 28 at 0:36
  • $\begingroup$ if you give me some background I may be able to provide some insight $\endgroup$ – George Dewhirst Nov 28 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.