This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
2answers
895 views

Entropy for three random variables [duplicate]

I'm just working through some information theory and entropy, and I've come into a bit of a problem. In many texts, it's easy to find the "chain rule" for entropy in two variables, and the "...
0
votes
2answers
40 views

Required bits to communicate a partial order?

Suppose that you have a ranking (i.e. a strict complete partial order) over $n$ different objects, so that the objects can be ordered as $a>b>\cdots>n$. You want to communicate the exact ...
2
votes
1answer
135 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
0
votes
0answers
48 views

Application of Jensen's Inequality. Correct?

Help would be appreciated. Consider $x \in (0,1)$ and $f(x)=x^2$ which is convex we want to show that $\mathbb{E}\Big[f(X)\Big] \geq f\Big[\mathbb{E}(X)\Big]$. Therefore, $\mathbb{E}\Big[X^{2}\Big] \...
0
votes
3answers
98 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
0
votes
1answer
37 views

Relative Entropy - Help please

I'm a bit stuck evaluating the relative entropy $\int_{}^{} f(\textbf{x}) \log \left(\tfrac{f(\textbf{x})}{g(\textbf{x})} \right) \mathrm{d}\textbf{x}$ (where f and g are two densities) in the case ...
1
vote
1answer
92 views

What's the maximum entropy for discrete distribution given mean and variance

I know for continuous distribution, given mean and variance, it's Normal distribution. I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance. I ...
1
vote
2answers
43 views

For any $n\ge2$ prove that $H(X_1,X_2,…,X_n)\ge\sum\limits_{i=n}^\mathbb{n}\ H(X_i|X_j , j \neq i)$

I am trying to figure this out and I am stuck. Any ideas? For any $n\ge2$ prove that $H(X_1,X_2,\ldots,X_n)\ge\sum\limits_{i=1}^\mathbb{n}\ H(X_i\mid X_j , \ j \neq i)$
1
vote
0answers
22 views

Maximum entropy distribution given second order marginals

Let $p(x,y,z)$ be a probability distribution over 3 variables (suppose them discrete, but it shouldn't matter). I know that the distribution with maximal entropy which preserves the first order ...
0
votes
1answer
68 views

Ratio between forward and reverse conditional probability

I have a probability distribution $p(Z | X)$ from which I can easily sample and compute the probability at every value for $Z$ and $X$. The inverse distribution $p(X | Z)$ however can be very complex ...
2
votes
2answers
401 views

How to prove H(X,Y) $\ge$ H(Z)?

I'm solving a problem from elements of information theory, 2nd. I got stuck by question(c) and actually, I've checked the answer, here it is: How to prove the inequality from the answer that is H(...
1
vote
1answer
38 views

Is there any differentiable function $f$ that approximates the “entropy” of a set of numbers $S$?

Where entropy is some measure of the degree of randomness/disorder in a given set of numbers: $S = \{a_1, a_2, ..., a_i\}$ For example, the set $S_{high} = \{4,0,2,5,8,3,7,2,5\}$ has a high degree of ...
2
votes
1answer
182 views

Coupon Collector Problem for Non-Uniform Coupons: On the number of missed Coupon

Suppose $\mathcal B=\{1,2,\ldots,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,\ldots,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$. The "...
2
votes
0answers
103 views

What is the variance of self-information (or surprisal)?

The self-information of an outcome $x_i$, or surprisal, is defined as: $$ I(x_i)=-\log P(x_i), $$ where $P$ means probability. This way, the Shannon entropy can be seen as the "average" or "expected" ...
0
votes
1answer
34 views

Entropy of $Y=bX$

If I have two random variables $Y$ and $U$ related as $Y=bU$, where $b>0$ is a constant and knowing that $\text{H}(x)$ represents the shannon entropy, such that: $$ \text{H}(x)=−\int \text{p}(x) \...
0
votes
0answers
33 views

Why is ln(p_i) not rounded down in theexpression for Shannon entropy?

Entropy supposedly " is the average amount of information contained in each message received"(Wikipedia: Entropy). However, to calculate the Shannon entropy for a finite sample, we have the sum over ...
1
vote
1answer
88 views

data processing inequality using non-deterministic functions

Generally data processing inequality says that the entropy cannot increase on applying a function f, or to be precise $H(f(X))\leq H(X)$ (also it is reversed if we know the function is k-to-1 so there ...
1
vote
1answer
183 views

Entropy of a uniform distribution

The entropy of a uniform distribution is $ ln(b-a)$. With $a=0$ and $b=1$ this reduces to zero. How come there is no uncertainty?
0
votes
0answers
25 views

Change of bases for entropy

From Cover and Thomas, Elements of Information Theory: Why isn't it: $ \log_b(p) = \frac{\log_a(p)}{\log_a(b)} $, so that $ H_a(X) $ is multiplied with $ \frac{1}{\log_b a} $?
2
votes
0answers
43 views

Finding the Legendre transform of an “entropy type” functional

I want to find the Legendre Transform of $$T(f) = \int_{\mathbb{R}^2} f \log \left(\frac{f}g{}\right) \, dx$$ on a set $H_M = \{ f: f \ge 0 \text{ and } \int_{\mathbb{R}^2} f =M\}$, where g is some ...
1
vote
0answers
109 views

Entropy calculation of Fibonacci distribution

For any positive integer $N$, consider the Fibonacci sequence $F_n$ of length $N$. Using $F_n$ we can define a Fibonacci discrete probability distribution as follows: $$p_N(n)=\frac{F_n}{\sum_{k=1}^N ...
1
vote
0answers
43 views

Estimate uncertainty of a function

I have a question about estimate the uncertainty of a function. I have a variable $x$. I assume that I can predict the variable $x$ is followed by a function $f(x)$ such as $$f(x)=\exp(-x^2)$$ Now, ...
7
votes
1answer
77 views

Structure of equientropic transformations

Given a probability vector $v=(v_1,\ldots,v_n)$ with $1\geq v_i\geq 0$ and $\sum_{i=1}^n v_i=1$ its entropy can be defined as: $$ H(v):=-\sum_{i=1}^nv_i\log v_i $$ I wonder what is known about ...
0
votes
0answers
17 views

A simpler way for an entropy inequality

I have to show that $\frac {1}{N}H(X_1,...,X_N)\le H(X_1)$. for a stationary stochastic process. I know that $H(X_1,...,X_N)=\Sigma _{i=1}^N H(X_i|X_1,...,X_{i-1})$. So far I have plugged that ...
0
votes
1answer
36 views

Mutual Information Entropy Inequality

I am trying to prove $H(x,y:z)>H(x:z)+H(y:z)$ and here is what I have. LHS: $=H(xy)-H(xy|z)=-\Sigma p(xy)lg(p(xy))+\Sigma p(xy|z)lg( \frac{p(xyz)}{p(z)})$ RHS: =$H(x)+H(z)-H(xz)+H(y)+H(z)-H(yz)=-\...
0
votes
0answers
73 views

How to calculate the mutual information between two outputs of Rayleigh fading channels

We have the two channels: $$X_{a,i} = H_{i}s_{i} +N_{a,i} \\ X_{b,i} = H_{i}s_{i} +N_{b,i} $$ for $1 \leq i \leq n$, where $H_i$ denotes the i.i.d. channel coefficient and is a zero-mean complex ...
0
votes
1answer
33 views

Relative Entropy decomposition reference

may I ask for some reference pointers? My bad as I got a classic case of losing my reference and thus unsure what I wrote was right or wrong. I tried looking my old references and internet and didn't ...
0
votes
2answers
50 views

Proving certain aspects of Entropy

I am trying to prove three properties of entropy. $1)$ $H(X|Y,Z)\le H(X|Y)$ $2)$ $H(X|Y,Z)\le H(X,Y)$ $3)$ $H(X,Y,Z)+H(Y)\le H(X,Y)+H(Y,Z)$ I have proved the third one, but it is based on part 1. ...
0
votes
1answer
51 views

Motivation for the binary entropy function

What is the motivation for the definition of the binary entropy function $H(x) = -p\log_2(p) - (1-p)\log_2(1-p)$? I understand that we want the entropy to be zero at $p = 0$ and $p = 1$ (no randomness)...
1
vote
1answer
118 views

Covering Number of Lipschitz Function Space

On page 12 of the slide deck here, the author gives an example where a lower bound (and upper bound, but I am particularly interested in the lower bound) on the $\epsilon$-covering number of a ...
2
votes
0answers
71 views

Limit of $\frac{1}{n}\log({n\choose np})$ without using Stirling's formula

I am trying to evaluate the following limit: $$ \forall p\in(0,1) ,\lim_{n\rightarrow \infty} \frac{1}{n}\log{n\choose \lfloor np \rfloor} =H(p),$$ where $\lfloor x\rfloor$ means the greatest integer ...
0
votes
1answer
60 views

Metric Entropy Upper Bounds

In the paper Information-Theoretic Determination of Minimax Rates of Convergence the authors present Theorem 3 as follows: If $M_2(\epsilon)$ is the $\ell_2$ packing entropy of a density class $\...
0
votes
2answers
83 views

Question regarding the Entropy of a probability mass function

I assume that the entropy, $E$, of a probability mass function (pmf), $p(X)$, of a discrete random variable, $X$, is computed as: $$\begin{align}\mathbb{E}(p(X)) &= -p(X = x_1)\log[p(X = x_1)]-p(...
2
votes
1answer
36 views

conditional entropies for identical distributions

Let me say I have two distributions $X$ and $Y$ which are identical, but they are not independent. Now if were to calculate the conditional entropies $H(X|Y)$ and $H(Y|X)$. Is calculating one joint ...
0
votes
0answers
85 views

Absolute value of difference between entropies (of two distributions)

I have the following inequality for the $L_1$ distance between two distributions $Q$, $Q^n$ on a finite set $B$: $$\|Q-Q^n|| < \frac{2|B|}{n}\leq \frac{C}{n} \leq \frac12 $$ Assuming $C\geq2|B|$, ...
2
votes
0answers
87 views

Shannon Entropy, prove $H(Wx)=H(x)+\log|\det W|$

I'm doing an essay on ICA (independent component analysis), and I could use some help. In essence, ICA is an algorithm that minimizes the entropy of $n$ $1$-dimensional random variables, but to show ...
1
vote
0answers
44 views

Entropy proerty

Let $a,b,c>0$ be distinct postive reals. Define four different probability distributions: $$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ $$\mathcal{P}_{bc}:P_{b,bc}=\frac{b}{b+c}=1-P_{c,...
1
vote
1answer
63 views

Understanding an application of Entropy

I'm struggling with the following exercise on entropy. Suppose that your friend Alice chooses a number $X$ uniformly at random from $[1,n]$, which she writes down using $\log n$ bits; you can assume ...
0
votes
1answer
75 views

Inverse of shannon entropy

The shannon entropy of a bit $(p,1-p)$ is $$H(p)=-p\log(p)-(1-p)\log(1-p)$$. This is a well behaved function that uniquley assigns each state (up to permutation of its elements, i.e. $(p,1-p)\...
1
vote
1answer
186 views

Entropy of floating number array

I am familiar with shanon's definition of entropy. $$ H(P) = - \sum_{i=1}^n p_i \cdot \log_2(\mathcal p_i) $$ I am today in the situation that I'd like to compute an entropy like function but for a ...
1
vote
1answer
64 views

Entropy Solution of $u_t+(u^2/2)_x=0$

Given the initial data $$ g(x)= \cases{ 1 & x< -1 \\ 0 & -1 < x< 0 \\ 2 & 0 < x< 1 \\ 0 & 1 < x \\ } $$ What is the entropy solution of $u_t+(u^2/2)_x=0$?
2
votes
2answers
82 views

Reference for entropy of the binomial distribution?

The Wikipedia page Binomial distribution says that the entropy of the Binomial(n,p) is $\frac{1}{2}\log_2\left(2\pi e n p (1-p)\right) + O\left(\frac{1}{n}\right)$. What is a reference (paper or ...
0
votes
0answers
65 views

Entropy of beta-expansion

We have the transformation $T: [0,1) \rightarrow [0,1)$ given by $Tx = \beta x \text{ mod } 1$ with $\beta = \frac{1+ \sqrt{5}}{2}$. Calculate the entropy $h_{\mu}(T)$ of $T$ wrt the invariant ...
3
votes
1answer
93 views

Shannon Entropy Continuity Constraint

I have the following problem: I want to find the probability density $p$ which maximizes the Shannon entropy \begin{equation} S := - \int_{x_b}^{x_c} dx ~ p(x) \log (p(x)) \end{equation} under the ...
0
votes
1answer
106 views

Shannon entropy on the simplex

The state of a trit $$\{p_1,p_2,p_3=1-p_1-p_2\}$$ can be represented as a triangular simplex. The centre of the simplex is the maximally mixed state $$m=\{\frac{1}{3},\frac{1}{3},\frac{1}{3}\}$$. And ...
0
votes
1answer
22 views

Greenberg-Hastings-Model: What kind of shift space is it?

I would like to read something about the entropy of the one-dimensional Greenberg-Hastings-Model - and I think maybe I can find something about that in the book "Symbolic Dynamics and Coding" - but I ...
0
votes
1answer
64 views

Is this formula a KL divergence?

As everyone knows KL divergence's formula is $KL(p||q) = \sum_{i=1}^{n}p(i)\log (p(i)/q(i))$. In the image, formula(9) is really calculate KL(X||($(UZ^TA^T)$)) , however i have no idea why there is "...
1
vote
0answers
27 views

Summation methods and entropy

I am aware of the theory of divergent series, but don't know much of it. If you have a text to recommend, I'd be glad to hear it. Suppose I have an infinite-dimensional probability vector $\mathbf{p} =...
1
vote
2answers
91 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & 1 & ...
2
votes
1answer
163 views

Is there a symmetric alternative to Kullback-Leibler divergence?

I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity ...