I am stuck with the following problem. Given the Gaussian mixture distribution $f(\cdot)$

$$ f(x) = \frac{a}{\sqrt{2\pi\sigma_1^2}}e^{-\frac{x^2}{2\sigma_1^2}}+\frac{1-a}{\sqrt{2\pi\sigma_2^2}}e^{-\frac{(x-\mu)^2}{2\sigma_2^2}} $$ where $a\in (0,0.5)$, $\mu>0$ and $\sigma_1 \neq \sigma_2$ so that $f(x)$ is skewed. The mixtures of $f(x)$, $$ f_1(x) = p\cdot f(x-c) + (1-p)\cdot f(x+c), $$ $$ f_2(x) = \frac{1}{2}\cdot f(x-c) +\frac{1}{2}\cdot f(x+c) $$ where $c$ is a constant. Prove that there exist a $p\in(0,1)$ so that the following inequality holds:

$$ -\int_{-\infty}^{\infty}f_1(x)\log f_1(x)dx>-\int_{-\infty}^{\infty}f_2(x)\log f_2(x)dx $$ where the LHS is the entropy of $f_1(x)$ and the RHS is the entropy of $f_2(x)$.Any suggestions? Thanks for your time in advance.

  • $\begingroup$ So we have the mixture of a mixture? Are you sure you got this right? What is $c$? $\endgroup$ – leonbloy Sep 23 '14 at 14:00
  • $\begingroup$ Hi, leonbloy. Yes, we have a mixture of a mixture. c is some constant in R. So $f_1$ and $f_2$ are four-term Gaussian mixture. The log-sum-exp problems are in general hard to solve. Do you have any ideas? $\endgroup$ – aniki_aishiteru Sep 23 '14 at 14:46

Define $$h(p) := -\int_{-\infty}^{\infty}f_1(x,p)\log f_1(x,p)\,dx $$ and show that for some $p\in(0,1)$, $$h(p) > h(1/2). $$

This can be done by showing that $h$ has a maximum at some $p \in(0,1)\backslash\{1/2\}$.


  1. Find $p$ such that $\frac{d}{dp}h(p) = 0$ and $\frac{d^2}{dp^2}h(p) < 0$.

  2. $\int_{-\infty}^{\infty}\left(\frac{\partial}{\partial\,p}f_1(x,p)\right)dx = 0.$

  3. $\frac{d}{dp}h(p) = -\int_{-\infty}^{\infty}\left(\frac{\partial}{\partial p}f_1(x,p)\right)\log f_1(x,p)\,dx $

  4. $\frac{\partial^2}{\partial p^2}f_1(x,p) = 0.$

  5. $\frac{d^2}{dp^2}h(p) = -\int_{-\infty}^{\infty}\left(\frac{\frac{\partial}{\partial p}f_1(x,p)}{f_1(x,p)} \right)^2f_1(x,p)\, dx.$

  • $\begingroup$ Hi r.e.s., I have tried this approach weeks ago. If I take the first derivative and let it equal to $0$, I get $\int (f(x-c)-f(x+c))\log (p f(x+c)+(1-p)f(x-c)) dx =0$ and I don't have any idea on how to find $p$ from here. Probably for some certain types of $f$ (some $f$ in the exponential family?), I can find a close form of $p$. Also if $f$ is symmetric, $p$ will be $\frac{1}{2}$. However, in the gaussian mixture case, I have no idea how to find $p$. $\endgroup$ – aniki_aishiteru Sep 28 '14 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.