Relation between the entropies $h(\sum w_i f_i)$, $h(\sum w_i g_i)$, when $h(f_i) \ge h(g_i)$?

Problem

$$f_i, g_i$$ are the probability density functions of the symmetric, unimodal distributions with the common center $$c_i$$. Assume the following:

• All $$g_i$$ are the translations of $$g_0$$ (whose center is $$0$$). Thus, $$h(g_i) = h(g_0), \forall i=1,\cdots,n$$.
• $$h(f_i) \ge h(g_0)$$, $$\forall i$$

where $$h(\xi)$$: differential entropy of the random variable following the density function $$\xi$$. Let's consider two mixture distributions with the shared weights, $$\displaystyle f=\sum_{i=1}^n w_i f_i$$, $$\displaystyle g=\sum_{i=1}^n w_i g_i$$, such that $$\displaystyle \sum_{i=1}^n w_i = 1$$.

Now, I want to prove or disprove my intuition here:

$$\displaystyle h(f) \ge h(g)$$

Try

I am literally stuck at the very first part: \begin{aligned} h(f) &= h\left(\sum_{i=1}^n w_i f_i\right) \\ &= - \int_{-\infty}^\infty \left(\sum_{i=1}^n w_i f_i (x) \right) \log \left(\sum_{i=1}^n w_i f_i (x) \right) dx \end{aligned}

Any suggestions will be welcomed. Many thanks!

• I encountered this problem when dealing with kernel density estimation (KDE) with kernel $$g_i$$.
• It would be nonetheless very helpful to know the results when we limit all the distributions $$f_i, g_i$$ to be Gaussian, which is usually the case in KDE.
• If $X$ has pdf $g$, then we can consider the $\{g_j\}_j$ to be the conditional pdfs of $X$ w.r.t. some other variable $Y$, where $Y$ has pmf given by the $\{w_j\}_j$. My intuition then says that $h(X)\approx h(Y)+\mathbb{E}_Y[h(X|Y)]$. Commented May 3, 2022 at 12:29
• @JacobManaker Interesting. Can you specify more? Commented May 3, 2022 at 13:11
• For what it's worth. the conjecture is false if the distributions are not restricted to be unimodal. Commented May 3, 2022 at 15:36
• @leonbloy Thanks! May I ask for a counterexample? Commented May 3, 2022 at 16:35
• I think your conjecture is false. Commented May 3, 2022 at 22:47

Not an answer, but a simple counterxample if the distributions are not restricted to be unimodal.

Take $$n=2$$ , $$c_1=0$$, $$c_2=2$$ and $$w_i=1/2$$.

Let $$u_{c,w}$$ denote the uniform distribution with center $$c$$ and width $$w$$ (that is, with support over $$[c-w/2,c+w/2]$$)

Let $$g_0 = u_{0,2}$$. Then $$g=u_{1,4}$$ and $$h(g)=\log 4 = 2$$ bits.

Let $$f_0= \frac{1}{2}(u_{-1,1}+u_{1,1})$$, with $$f_i$$ given by the same translations as $$g_i$$. Then $$h(f_i)=h(g_i)=1$$ and

$$f= \frac{1}{4}u_{-1,1}+\frac{1}{2}u_{1,1}+\frac{1}{4}u_{3,1}$$

with $$h(f) = h(u_{1,1}) + h_d(\frac{1}{4},\frac{1}{2},\frac{1}{4})= 0 + \frac{3}{2} = 1.5$$ bits

It suffices to make a small perturbation to $$f_i$$ (change its width from $$1$$ to $$1+\epsilon$$) to attain $$h(f_i)>h(g_i)$$ and keep $$h(f)

For the unimodal case.

Take $$n=2$$ , $$c_1=0$$, $$c_2=2$$ and $$w_i=1/2$$, as before, and let $$g$$ be the same as before.

Let $$f_0= \beta u_{-1,1} + \alpha u_{0,1} + \beta u_{0,1}$$ with $$0<\alpha <1$$ and $$\beta = \frac{1-\alpha}{2}<\alpha$$. Then $$h(f_i)= h_d(\beta,\alpha,\beta)$$. And

$$f= \frac{\beta}{2} u_{-1,1} + \frac{\alpha}{2} u_{0,1} + {\beta} u_{1,1} + \frac{\alpha}{2} u_{2,1} + \frac{\beta}{2} u_{3,1}$$

with $$h(f)=h_d(\frac{\beta}{2}, \frac{\alpha}{2}, \beta, \frac{\alpha}{2} ,\frac{\beta}{2})$$

where $$h_d$$ denotes the discrete entropy. By choosing $$\alpha = \frac{3}{4}$$ we get $$h(f_i)=1.061278 > 1 = h(g_i)$$ but $$h(f)=1.936278 < 2 = h(g)$$

Of course, $$g_0$$ and $$f_0$$ need only an arbitrarily small perturbation to make them strictly unimodal.