# Prove there exist a $p$ so that the inequality holds

I am stuck with the following problem. Given the Gaussian mixture distribution $f(\cdot)$

$$f(x) = \frac{a}{\sqrt{2\pi\sigma_1^2}}e^{-\frac{x^2}{2\sigma_1^2}}+\frac{1-a}{\sqrt{2\pi\sigma_2^2}}e^{-\frac{(x-\mu)^2}{2\sigma_2^2}}$$ where $a\in (0,0.5)$, $\mu>0$ and $\sigma_1 \neq \sigma_2$ so that $f(x)$ is skewed. The mixtures of $f(x)$, $$f_1(x) = p\cdot f(x-c) + (1-p)\cdot f(x+c),$$ $$f_2(x) = \frac{1}{2}\cdot f(x-c) +\frac{1}{2}\cdot f(x+c)$$ where $c$ is a constant. Prove that there exist a $p\in(0,1)$ so that the following inequality holds:

$$-\int_{-\infty}^{\infty}f_1(x)\log f_1(x)dx>-\int_{-\infty}^{\infty}f_2(x)\log f_2(x)dx$$ where the LHS is the entropy of $f_1(x)$ and the RHS is the entropy of $f_2(x)$.Any suggestions? Thanks for your time in advance.

• So we have the mixture of a mixture? Are you sure you got this right? What is $c$? – leonbloy Sep 23 '14 at 14:00
• Hi, leonbloy. Yes, we have a mixture of a mixture. c is some constant in R. So $f_1$ and $f_2$ are four-term Gaussian mixture. The log-sum-exp problems are in general hard to solve. Do you have any ideas? – aniki_aishiteru Sep 23 '14 at 14:46

Define $$h(p) := -\int_{-\infty}^{\infty}f_1(x,p)\log f_1(x,p)\,dx$$ and show that for some $p\in(0,1)$, $$h(p) > h(1/2).$$

This can be done by showing that $h$ has a maximum at some $p \in(0,1)\backslash\{1/2\}$.

Hints:

1. Find $p$ such that $\frac{d}{dp}h(p) = 0$ and $\frac{d^2}{dp^2}h(p) < 0$.

2. $\int_{-\infty}^{\infty}\left(\frac{\partial}{\partial\,p}f_1(x,p)\right)dx = 0.$

3. $\frac{d}{dp}h(p) = -\int_{-\infty}^{\infty}\left(\frac{\partial}{\partial p}f_1(x,p)\right)\log f_1(x,p)\,dx$

4. $\frac{\partial^2}{\partial p^2}f_1(x,p) = 0.$

5. $\frac{d^2}{dp^2}h(p) = -\int_{-\infty}^{\infty}\left(\frac{\frac{\partial}{\partial p}f_1(x,p)}{f_1(x,p)} \right)^2f_1(x,p)\, dx.$

• Hi r.e.s., I have tried this approach weeks ago. If I take the first derivative and let it equal to $0$, I get $\int (f(x-c)-f(x+c))\log (p f(x+c)+(1-p)f(x-c)) dx =0$ and I don't have any idea on how to find $p$ from here. Probably for some certain types of $f$ (some $f$ in the exponential family?), I can find a close form of $p$. Also if $f$ is symmetric, $p$ will be $\frac{1}{2}$. However, in the gaussian mixture case, I have no idea how to find $p$. – aniki_aishiteru Sep 28 '14 at 17:27