1
vote
1answer
103 views

Prove there exist a $p$ so that the inequality holds

I am stuck with the following problem. Given the Gaussian mixture distribution $f(\cdot)$ $$ f(x) = ...
1
vote
2answers
105 views

Upper bound on the entropy of a sum two random variables

Let $X$ be a random variable such that $|X| \leq A$ almost surely, for some $A > 0$. Let $Z$ be independent of $X$ such that $Z \sim {\cal N}(0, N)$. My question is: How large can the entropy ...
0
votes
0answers
139 views

Proof of Mrs. Gerber's Lemma Using Convexity and Jensen's Inequality

Can anyone give a proof of the Mrs. Gerber’s Lemma for the scalar case: $$ \ H^{-1}(H(Y|U)) \ge H^{-1}(H(X|U))*p $$ where $$ \ a*b = a(1-b) + b(1-a) $$ $$ \ X\ ,\ Y $$ are binary random ...
1
vote
1answer
123 views

Bounding mutual information given ROC curve statistics

When evaluating a binary classifier, the basic data are as in this contingency table, where rows represent groundtruth value and columns represent the estimated value: $$ \begin{matrix} & + ...
5
votes
1answer
143 views

A tight lower bound for the entropy of the XOR of two random variables

Let $U$ be the uniform random variable over $n$-bit binary strings, and let $X$ be another random variable that is dependent on $U$ and ranges over $n$-bit binary strings. Assuming $I(X;U) \le ...
6
votes
1answer
245 views

Lower bound on binomial coefficient

I encountered the following claim $$\frac{1}{n+1}2^{nH_2(k/n)} \le \binom{n}{k} \le 2^{nH_2(k/n)}$$ where $H_2$ is the binary entropy function. The upper bound is rather well known but how does one ...
0
votes
0answers
636 views

Binary symmetric channel capacity or mutual information inequality

I proved that I(X,Y) <= 1 - H(p) to the following way: How can I prove if I start in that way I(X,Y) = H(X) - H(X|Y), I ...
5
votes
1answer
1k views

Proof of Pinsker's inequality.

How to prove the following known (Pinsker's) inequality? For two strictly positive sequences $(p_i)^n_{i=l}$ and $(q_i)^n_{i=l}$ with $\sum_{i=1}^np_i=\sum_{i=1}^nq_i=1$ one has ...
2
votes
2answers
164 views

Proving Asymptotic Equipartition Property for Gaussian r.v.'s using the Chernoff Bound

I just learned about Chernoff Bounds and am wondering if one can prove the Asymptotic Equipartition Property using them instead of the Weak Law of Large Numbers (which is the consequence of the ...
1
vote
1answer
122 views

Find the maximum of this binary entropy look-alike function

Let $b,c \in (0,1)$ be such that $b+c<1.$ Define the following function for $p \in (0,1) :$ $$ I(p;b,c):=(b+c)[p \log\frac{1}{p}+(1-p)\log\frac{1}{1-p}]-cH(p;b,c)-bH(p;c,b)$$ where ...
3
votes
1answer
314 views

Lower bound on the entropy of the mixture of two zero-mean Gaussians

I am wondering if someone knows of a lower bound on the differential entropy of a mixture of two zero-mean Gaussians: $$h(X)=-\int_{-\infty}^{\infty} f_X(x)\log f_X(x)dx$$ where ...
4
votes
1answer
528 views

Upper bound for variance of an (arbitrary) zero-mean random variable $X$ given distance between it and a known random variable $Y$

I have a zero-mean Gaussian random variable $Y\sim\mathcal{N}(0,\sigma^2_X)$ with known variance $\sigma_X^2$. I also have a zero-mean random variable $X$, which may be dependent on $Y$ (though, I ...
4
votes
1answer
766 views

Understanding the relationship of the $L^1$ norm to the total variation distance of probability measures, and the variance bound on it

I am trying to find a bound for variance of an arbitrary distribution $f_Y$ given a bound of a Kullback-Leiber divergence from a zero-mean Gaussian to $f_Y$, as I've explained in this related ...
4
votes
0answers
296 views

Inequalities involving the probability density function and variance

I am wondering whether anyone knows of any any inequalities involving the probability density function of an unknown distribution (as opposed to the cumulative distribution function) and its known ...
22
votes
2answers
1k views

An information theory inequality which relates to Shannon Entropy

For $a_1,...,a_n,b_1,...,b_n>0,\quad$ define $a:=\sum a_i,\ b:=\sum b_i,\ s:=\sum \sqrt{a_ib_i}$. Is the following inequality true?: $${\frac{\Bigl(\prod a_i^{a_i}\Bigr)^\frac1a}a \cdot ...