$f_i, g_i$ are the probability density functions of the symmetric, unimodal distributions with the common center $c_i$. Assume the following:

  • All $g_i$ are the translations of $g_0$ (whose center is $0$). Thus, $h(g_i) = h(g_0), \forall i=1,\cdots,n$.
  • $h(f_i) \ge h(g_0)$, $\forall i$

where $h(\xi)$: differential entropy of the random variable following the density function $\xi$. Let's consider two mixture distributions with the shared weights, $\displaystyle f=\sum_{i=1}^n w_i f_i$, $\displaystyle g=\sum_{i=1}^n w_i g_i$, such that $\displaystyle \sum_{i=1}^n w_i = 1$.

Now, I want to prove or disprove my intuition here:

$\displaystyle h(f) \ge h(g)$


I am literally stuck at the very first part: $$ \begin{aligned} h(f) &= h\left(\sum_{i=1}^n w_i f_i\right) \\ &= - \int_{-\infty}^\infty \left(\sum_{i=1}^n w_i f_i (x) \right) \log \left(\sum_{i=1}^n w_i f_i (x) \right) dx \end{aligned} $$

Any suggestions will be welcomed. Many thanks!

Additional remarks

  • I encountered this problem when dealing with kernel density estimation (KDE) with kernel $g_i$.
  • It would be nonetheless very helpful to know the results when we limit all the distributions $f_i, g_i$ to be Gaussian, which is usually the case in KDE.
  • 1
    $\begingroup$ If $X$ has pdf $g$, then we can consider the $\{g_j\}_j$ to be the conditional pdfs of $X$ w.r.t. some other variable $Y$, where $Y$ has pmf given by the $\{w_j\}_j$. My intuition then says that $h(X)\approx h(Y)+\mathbb{E}_Y[h(X|Y)]$. $\endgroup$ Commented May 3, 2022 at 12:29
  • $\begingroup$ @JacobManaker Interesting. Can you specify more? $\endgroup$
    – Moreblue
    Commented May 3, 2022 at 13:11
  • 1
    $\begingroup$ For what it's worth. the conjecture is false if the distributions are not restricted to be unimodal. $\endgroup$
    – leonbloy
    Commented May 3, 2022 at 15:36
  • $\begingroup$ @leonbloy Thanks! May I ask for a counterexample? $\endgroup$
    – Moreblue
    Commented May 3, 2022 at 16:35
  • 1
    $\begingroup$ I think your conjecture is false. $\endgroup$
    – leonbloy
    Commented May 3, 2022 at 22:47

1 Answer 1


Not an answer, but a simple counterxample if the distributions are not restricted to be unimodal.

Take $n=2$ , $c_1=0$, $c_2=2$ and $w_i=1/2$.

Let $u_{c,w}$ denote the uniform distribution with center $c$ and width $w$ (that is, with support over $[c-w/2,c+w/2]$)

Let $g_0 = u_{0,2}$. Then $g=u_{1,4}$ and $h(g)=\log 4 = 2$ bits.

Let $f_0= \frac{1}{2}(u_{-1,1}+u_{1,1})$, with $f_i$ given by the same translations as $g_i$. Then $h(f_i)=h(g_i)=1$ and

$$f= \frac{1}{4}u_{-1,1}+\frac{1}{2}u_{1,1}+\frac{1}{4}u_{3,1}$$

with $h(f) = h(u_{1,1}) + h_d(\frac{1}{4},\frac{1}{2},\frac{1}{4})= 0 + \frac{3}{2} = 1.5$ bits

It suffices to make a small perturbation to $f_i$ (change its width from $1$ to $1+\epsilon$) to attain $h(f_i)>h(g_i)$ and keep $h(f)<h(g)$

For the unimodal case.

Take $n=2$ , $c_1=0$, $c_2=2$ and $w_i=1/2$, as before, and let $g$ be the same as before.

Let $$f_0= \beta u_{-1,1} + \alpha u_{0,1} + \beta u_{0,1} $$ with $ 0<\alpha <1$ and $\beta = \frac{1-\alpha}{2}<\alpha$. Then $h(f_i)= h_d(\beta,\alpha,\beta)$. And

$$f= \frac{\beta}{2} u_{-1,1} + \frac{\alpha}{2} u_{0,1} + {\beta} u_{1,1} + \frac{\alpha}{2} u_{2,1} + \frac{\beta}{2} u_{3,1}$$

with $h(f)=h_d(\frac{\beta}{2}, \frac{\alpha}{2}, \beta, \frac{\alpha}{2} ,\frac{\beta}{2})$

where $h_d$ denotes the discrete entropy. By choosing $\alpha = \frac{3}{4}$ we get $h(f_i)=1.061278 > 1 = h(g_i)$ but $h(f)=1.936278 < 2 = h(g)$

Of course, $g_0$ and $f_0$ need only an arbitrarily small perturbation to make them strictly unimodal.

enter image description here


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .