3
$\begingroup$

Problem

$f_i, g_i$ are the probability density functions of the symmetric, unimodal distributions with the common center $c_i$. Assume the following:

  • All $g_i$ are the translations of $g_0$ (whose center is $0$). Thus, $h(g_i) = h(g_0), \forall i=1,\cdots,n$.
  • $h(f_i) \ge h(g_0)$, $\forall i$

where $h(\xi)$: differential entropy of the random variable following the density function $\xi$. Let's consider two mixture distributions with the shared weights, $\displaystyle f=\sum_{i=1}^n w_i f_i$, $\displaystyle g=\sum_{i=1}^n w_i g_i$, such that $\displaystyle \sum_{i=1}^n w_i = 1$.

Now, I want to prove or disprove my intuition here:

$\displaystyle h(f) \ge h(g)$


Try

I am literally stuck at the very first part: $$ \begin{aligned} h(f) &= h\left(\sum_{i=1}^n w_i f_i\right) \\ &= - \int_{-\infty}^\infty \left(\sum_{i=1}^n w_i f_i (x) \right) \log \left(\sum_{i=1}^n w_i f_i (x) \right) dx \end{aligned} $$

Any suggestions will be welcomed. Many thanks!


Additional remarks

  • I encountered this problem when dealing with kernel density estimation (KDE) with kernel $g_i$.
  • It would be nonetheless very helpful to know the results when we limit all the distributions $f_i, g_i$ to be Gaussian, which is usually the case in KDE.
$\endgroup$
6
  • 1
    $\begingroup$ If $X$ has pdf $g$, then we can consider the $\{g_j\}_j$ to be the conditional pdfs of $X$ w.r.t. some other variable $Y$, where $Y$ has pmf given by the $\{w_j\}_j$. My intuition then says that $h(X)\approx h(Y)+\mathbb{E}_Y[h(X|Y)]$. $\endgroup$ Commented May 3, 2022 at 12:29
  • $\begingroup$ @JacobManaker Interesting. Can you specify more? $\endgroup$
    – Moreblue
    Commented May 3, 2022 at 13:11
  • 1
    $\begingroup$ For what it's worth. the conjecture is false if the distributions are not restricted to be unimodal. $\endgroup$
    – leonbloy
    Commented May 3, 2022 at 15:36
  • $\begingroup$ @leonbloy Thanks! May I ask for a counterexample? $\endgroup$
    – Moreblue
    Commented May 3, 2022 at 16:35
  • 1
    $\begingroup$ I think your conjecture is false. $\endgroup$
    – leonbloy
    Commented May 3, 2022 at 22:47

1 Answer 1

2
$\begingroup$

Not an answer, but a simple counterxample if the distributions are not restricted to be unimodal.

Take $n=2$ , $c_1=0$, $c_2=2$ and $w_i=1/2$.

Let $u_{c,w}$ denote the uniform distribution with center $c$ and width $w$ (that is, with support over $[c-w/2,c+w/2]$)

Let $g_0 = u_{0,2}$. Then $g=u_{1,4}$ and $h(g)=\log 4 = 2$ bits.

Let $f_0= \frac{1}{2}(u_{-1,1}+u_{1,1})$, with $f_i$ given by the same translations as $g_i$. Then $h(f_i)=h(g_i)=1$ and

$$f= \frac{1}{4}u_{-1,1}+\frac{1}{2}u_{1,1}+\frac{1}{4}u_{3,1}$$

with $h(f) = h(u_{1,1}) + h_d(\frac{1}{4},\frac{1}{2},\frac{1}{4})= 0 + \frac{3}{2} = 1.5$ bits

It suffices to make a small perturbation to $f_i$ (change its width from $1$ to $1+\epsilon$) to attain $h(f_i)>h(g_i)$ and keep $h(f)<h(g)$


For the unimodal case.

Take $n=2$ , $c_1=0$, $c_2=2$ and $w_i=1/2$, as before, and let $g$ be the same as before.

Let $$f_0= \beta u_{-1,1} + \alpha u_{0,1} + \beta u_{0,1} $$ with $ 0<\alpha <1$ and $\beta = \frac{1-\alpha}{2}<\alpha$. Then $h(f_i)= h_d(\beta,\alpha,\beta)$. And

$$f= \frac{\beta}{2} u_{-1,1} + \frac{\alpha}{2} u_{0,1} + {\beta} u_{1,1} + \frac{\alpha}{2} u_{2,1} + \frac{\beta}{2} u_{3,1}$$

with $h(f)=h_d(\frac{\beta}{2}, \frac{\alpha}{2}, \beta, \frac{\alpha}{2} ,\frac{\beta}{2})$

where $h_d$ denotes the discrete entropy. By choosing $\alpha = \frac{3}{4}$ we get $h(f_i)=1.061278 > 1 = h(g_i)$ but $h(f)=1.936278 < 2 = h(g)$

Of course, $g_0$ and $f_0$ need only an arbitrarily small perturbation to make them strictly unimodal.

enter image description here

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .