0
votes
0answers
16 views

Hypercontractivity of Markov Operator

I have been reading a paper by Ahlswede and Gacs on hypercontractivity of Markov operator (see here 1) and its application to information theory. To be honest, I could not fully understand the ...
0
votes
0answers
46 views

Special case of Kullback-Leibler additivity

I have three random variables $X,Y,Z$. If $(X,Z)$ are an independent pair and $(Y,Z)$ are an independent pair, then the additive property of the Kullback-Leibler divergence says $K(X,Z|Y,Z) = K(X|Y) ...
3
votes
1answer
50 views

Upper Bound on Mutual Information

I am interested in an upper bound on mutual information that I have been encountering frequently in the statistics and probability literature. I have yet to see the "purest" form of the inequality, so ...
2
votes
0answers
34 views

Mutual Information for Gaussian Process (and also Fano's Inequality)

According to this presentation: Bounding Gaussian Process Information Gain we have a closed-form expression for the information gain as follows: $$ I\left(\vec{y} \mid f\right) = \frac{1}{2} \log\det ...
0
votes
1answer
42 views

Using binary entropy function to approximate log(N choose K)

I am not a mathematician and struggling with the exercises while reading this book Information Theory, Inference and Learning Algorithms. The author introduced the binary entropy function at the ...
0
votes
3answers
140 views

Similarity between two probability distribution

I am not sure how to put the question. I am not even sure if this question makes sense at all. I know that the similarity of two discrete (or continuous) distributions can be quantified by ...
0
votes
1answer
34 views

A question on Markov chain

Suppose for two random variables $X$ and $Y$ we have $X\perp\!\!\!\perp Y$ and also assume that three random variables $X$, $Y$ and $Z$ form the following Markov chain: $X\to Z\to Y$. Do these two ...
3
votes
0answers
59 views

Doubts in Bayes' Theorem

I meet one problem on the probability and statistic theory. "Assume given the probability spaces $(X,S,\mu_i)$, $i=1,2$, and the probability space $(X,S,\lambda)$. And there exsit functions ...
2
votes
2answers
43 views

Does X|Y = X formally, in the sense of RVs?

In Cover and Thomas' "Elements of Information Theory", the joint entropy $H(X,Y)$ is defined, but they state that this definition is nothing new if we consider that it is the entropy of a single ...
2
votes
2answers
81 views

Bounding second moment of entropy

Entropy is defined as $E(-\log(P(x))$. We know it is bounded by $\log(r)$ when $r$ is the size of alphabet. Defining the second moment as $E(\log^2(P(x))$, how to show it is bounded?
2
votes
3answers
59 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
1
vote
1answer
53 views

Shannon Entropy Minimization

The Shannon Entropy for an observation is given by $ -x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that ...
1
vote
1answer
52 views

Jensen's inequality for countable probability space

One form of Jensen's inequality for the finite case, tells us that $$ \sum_{x \in X} p(x) \log q(x) \leq \log\sum_{x \in X} p(x) \cdot q(x) $$ For positive p(x), and $\sum_{x \in X} p(x) = 1$, ...
0
votes
2answers
85 views

Are marginal densities always greater than the corresponding joint density?

I.e. if $\mathbb P\left(x,y\right)$ is a joint density function, and $\mathbb P\left(y\right)$ is a marginal distribution is it always true that: $\mathbb P\left(x,y\right)\leq \mathbb ...
0
votes
1answer
265 views

Is there any software package to calculate the entropy, information content, mutual information, etc?

Provided a p.f. of a discrete random variable, or a joint p.f. for several random variable, is there any software package to calculate the entropy, joint entropy, information content, mutual ...
5
votes
3answers
223 views

What does the -log[P(X)] mean in the calculation of entropy?

The entropy (self information) of a discrete random variable X is calculated as: $$ H(x)=E(-log[P(X)]) $$ What does the -log[P(X)] mean? It seems to be something like ""the self information of each ...
2
votes
0answers
58 views

Finding the mean and the variance of a martingale using concentration inequalities

I am trying to find the mean and the variance of a martingale defined as the maximized likelihood ratios over some finite parameter space. The way I want to do this is through Azuma's inequality (or ...
0
votes
0answers
115 views

Proof of Mrs. Gerber's Lemma Using Convexity and Jensen's Inequality

Can anyone give a proof of the Mrs. Gerber’s Lemma for the scalar case: $$ \ H^{-1}(H(Y|U)) \ge H^{-1}(H(X|U))*p $$ where $$ \ a*b = a(1-b) + b(1-a) $$ $$ \ X\ ,\ Y $$ are binary random ...
2
votes
0answers
37 views

Kullback–Leibler divergence with elements that are $0$

I have a problem that I need to calculate Kullback–Leibler divergence, but the problem is that I have some elements that are $0$. Is there a way that I'm able to deal with situations like that? I know ...
0
votes
1answer
28 views

KL-divergence with zero probability?

Suppose $P(X=1)=P(X=2)=1/2$ and $P(Y=1)=1$. Then $$D(Y||X)=\log\left(\frac{1}{1/2}\right)+0\cdot\log\left(\frac{0}{1/2}\right).$$ If we consider ...
1
vote
0answers
31 views

How to define a utility of an information source?

This is a more specific (and, hopefully, clearer version of a previous question). The utility of discovering the value of a random variable $X$ can be defined to be its information content. When ...
0
votes
0answers
35 views

How to define “compound entropy”

Entropy measures the "surprise" one experiences when uncovering a the actual value of a random variable as $$-\sum_i p_i \log_2 p_i$$ E.g., if we observe Red 8 ...
1
vote
1answer
70 views

Mutual information and additivity under independence?

So I've been trying to figure this out since I saw it quoted in a paper. Suppose $y$ and $z$ are two independent variables. Is it true then that $I(x;y) + I(x;z) \leq I(x:y,z)$? My intuition is that ...
1
vote
0answers
24 views

Does this quantity have a meaning or relation to something else?

$X$ is a discrete random variable taking values in $\{0,1,2,3,\ldots\}$ with a probability mass function $p_X(n)$. Let $$U_k(X)=\sum_{n=k}^\infty\sqrt{np_X(n)p_X(n-k)}$$ where $k$ is a positive ...
1
vote
0answers
34 views

Independent transformation of probability measures

I have a pair of dependent random variable $(\theta, X)$ where $\theta\in K$ for a compact subset $K\subset\mathbb{R}$ and $X\in\mathbb{R}^d$ with marginals $P_{\theta}$ and $P_X$. I want to ...
2
votes
0answers
102 views

Explanation of Radon-Nikodym derivates wrt to probabilities

I am currently working in communications, where a lot of work is done via probability calculations (densities and such). As I am not a mathematician, I do have a quite hard time understanding one ...
1
vote
0answers
68 views

what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
0
votes
2answers
49 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
2
votes
1answer
221 views

Conditional Independence and Mutual information

I have a question concerning conditional independence. According to wikipedia (yes, maybe not the best source) two random variables are conditionally independent given a third if $$p(x,y|z) = ...
1
vote
1answer
37 views

Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$

Let $X,Y, and \space Z$ be random variables. Let $A$ be a subset of $U$ such that $p(X \in U)=1$ Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$ Do these two expressions represent the same thing? ...
1
vote
1answer
391 views

Is maximizing entropy equivalent to minimizing the defined variance?

Assume there is multi-set of some integers : $D = \{a_1,a_2,\cdots,a_{N-1}\}$ such that $\sum_i a_i = A$ we can build a discrete probability distribution by dividing elements of set by $A$, i.e. $p_i ...
2
votes
1answer
83 views

Transformation of mutual information to probability distribution

Given the upper bound for mutual information of random variables $X$ and $Y$, $I(X;Y)\leq L$, what can we say about their joint distribution? I mean for example if $L=0$, then we know $p_{XY}(A\cap ...
1
vote
0answers
125 views

KL divergence of multinomial distribution

Consider $q(x)$ be a Multinomial distribution over $\{1, \ldots, k\}$ with parameters $\{\theta_1,\ldots, \theta_k\}$. And p(x) over $\{1,\ldots, k\}$ with distribution $p(x)=\frac{1}{k}$. Then what ...
1
vote
1answer
219 views

Is maximizing the Shannon differential entropy equivalent to minimizing the predictability and/or minimizing the maximum density?

For a real-valued, 1-dimensional, continuous random variable X with density f(x), I am trying to determine if maximizing the Shannon differential entropy of f(x) is mathematically equivalent to ...
3
votes
0answers
53 views

Is there a conditional version of the asymptotic equipartition property?

Let $X_i$ be independent random variables with $\operatorname{Pr}(X_i = x) = p_x$, and let $F_n$ be the empirical frequency distribution of $X_1, \ldots, X_n$: that is, $(F_n)_x$ For any frequency ...
0
votes
0answers
70 views

Intregral of exponential of Shannon Entropy Function

Here I am going to ask a similar question as rde asked , that is what is the integral of exponential of entropy function. That is what is the value of $F[H(x)]=\int_{-1}^{+1} e^{ikH(f(x^2))} dx$ ...
2
votes
2answers
117 views

Proving an asymptotic property regard the fraction of ‘1’ and ‘0’ in binary sequences

Consider the set of sequences of zeroes and ones of length $N$ with $k$ ones (or, $Np$ ones where $p = k/N$). We draw randomly and uniformly a sequence from this set. I want to show that with ...
0
votes
0answers
577 views

Binary symmetric channel capacity or mutual information inequality

I proved that I(X,Y) <= 1 - H(p) to the following way: How can I prove if I start in that way I(X,Y) = H(X) - H(X|Y), I ...
9
votes
2answers
588 views

In what sense is the Jeffreys prior invariant?

I've been trying to understand the motivation for the use of the Jeffreys prior in Bayesian statistics. Most texts I've read online make some comment to the effect that the Jeffreys prior is ...
6
votes
4answers
287 views

Is probability objective?

As we know, probability is a measure of events. However, is it an objectively attribute of events, or just an illusion in ones' mind? For example, suppose that there is an empty black box with an ...
1
vote
1answer
162 views

Maximally entropy preserving irreversible functions. (CS related)

The topic/problem is related to hashing for data structures used in programming, but I seek formal treatment. I hope that by studying the problem I will be enlightened of the fundamental limitations ...
1
vote
1answer
252 views

Explicit examples of smooth entropy computation

Smooth classic entropies generalize the standard notions of entropy. This smoothing stands for a minimization/maximization over all events $\Omega$ such that $p(\Omega)\geq 1-\varepsilon$ for a given ...
5
votes
1answer
988 views

Proof of Pinsker's inequality.

How to prove the following known (Pinsker's) inequality? For two strictly positive sequences $(p_i)^n_{i=l}$ and $(q_i)^n_{i=l}$ with $\sum_{i=1}^np_i=\sum_{i=1}^nq_i=1$ one has ...
4
votes
3answers
157 views

simulating a fair random process with an unfair one.

Let's say I have a stochastic process that outputs $1$ or $0$ with probability $p$ or $1-p$ respectively, $p\neq 1/2$. Let's assume this is a repeatable iid process. So I can generate $X_1,X_2\dots$ ...
4
votes
1answer
206 views

What is the general context for entropy (information theory)?

From Wikipedia: Let $X$ be a random variable with a probability density function $f$ whose support is a set $\mathbb{X}$. The differential entropy $h(f)$ is defined as $$ h(f) = ...
5
votes
1answer
224 views

measure of information

We know that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon's source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length ...
1
vote
1answer
112 views

Question about Mutual Information

I am learning about mutual information, and am confused about one of the definitions. Mutual information is defined as $ I(X;Y) = H(X) - H(X | Y) $ where, $$ H(X) = \sum_{x} p(x) \log ...
3
votes
1answer
288 views

Lower bound on the entropy of the mixture of two zero-mean Gaussians

I am wondering if someone knows of a lower bound on the differential entropy of a mixture of two zero-mean Gaussians: $$h(X)=-\int_{-\infty}^{\infty} f_X(x)\log f_X(x)dx$$ where ...
4
votes
1answer
680 views

What is the relationship of $\mathcal{L}_1$ (total variation) distance to hypothesis testing?

Kullback-Leibler divergence (a.k.a. relative entropy) has a nice property in hypothesis testing: given some observed measurement $m\in \mathcal{Q}$, and two probability distributions $P_0$ and $P_1$ ...
4
votes
1answer
517 views

Upper bound for variance of an (arbitrary) zero-mean random variable $X$ given distance between it and a known random variable $Y$

I have a zero-mean Gaussian random variable $Y\sim\mathcal{N}(0,\sigma^2_X)$ with known variance $\sigma_X^2$. I also have a zero-mean random variable $X$, which may be dependent on $Y$ (though, I ...