This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
1answer
73 views

What's the maximum entropy for discrete distribution given mean and variance

I know for continuous distribution, given mean and variance, it's Normal distribution. I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance. I ...
1
vote
2answers
41 views

For any $n\ge2$ prove that $H(X_1,X_2,…,X_n)\ge\sum\limits_{i=n}^\mathbb{n}\ H(X_i|X_j , j \neq i)$

I am trying to figure this out and I am stuck. Any ideas? For any $n\ge2$ prove that $H(X_1,X_2,\ldots,X_n)\ge\sum\limits_{i=1}^\mathbb{n}\ H(X_i\mid X_j , \ j \neq i)$
0
votes
0answers
18 views

Why differential entropy is called “differential”

The extension of the concept of entropy to continuous random variable is sometimes called continuous entropy, which makes sense, but is also often called "differential entropy". Do you know why ?
0
votes
0answers
22 views

Does there exist an uncertainty (entropy) monotonic pmf combination rule?

Assume that I have two probability mass functions (pmf's): $p:=[p_1, p_2, p_3]$ and $q:=[q_1 ,q_2, q_3]$. Further, I assume that the uncertainty of these pmf's is quantified by the Renyi quadratic ...
1
vote
0answers
21 views

Maximum entropy distribution given second order marginals

Let $p(x,y,z)$ be a probability distribution over 3 variables (suppose them discrete, but it shouldn't matter). I know that the distribution with maximal entropy which preserves the first order ...
0
votes
1answer
59 views

Ratio between forward and reverse conditional probability

I have a probability distribution $p(Z | X)$ from which I can easily sample and compute the probability at every value for $Z$ and $X$. The inverse distribution $p(X | Z)$ however can be very complex ...
2
votes
2answers
293 views

How to prove H(X,Y) $\ge$ H(Z)?

I'm solving a problem from elements of information theory, 2nd. I got stuck by question(c) and actually, I've checked the answer, here it is: How to prove the inequality from the answer that is ...
0
votes
0answers
37 views

Maximum of a function relatred to relative entropy of Gaussian Mixture Distribution

If we have the following functions \begin{equation} g(x)=\sum_{1}^{T}\frac{\epsilon_{i}}{2\pi\sigma_{i}^2}e^{-\frac{|\mathbf{x}|^2}{2\sigma_{i}^2}} \end{equation} \begin{equation} ...
1
vote
1answer
34 views

Is there any differentiable function $f$ that approximates the “entropy” of a set of numbers $S$?

Where entropy is some measure of the degree of randomness/disorder in a given set of numbers: $S = \{a_1, a_2, ..., a_i\}$ For example, the set $S_{high} = \{4,0,2,5,8,3,7,2,5\}$ has a high degree of ...
2
votes
1answer
152 views

Coupon Collector Problem for Non-Uniform Coupons: On the number of missed Coupon

Suppose $\mathcal B=\{1,2,\ldots,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,\ldots,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$. The ...
2
votes
0answers
81 views

What is the variance of self-information (or surprisal)?

The self-information of an outcome $x_i$, or surprisal, is defined as: $$ I(x_i)=-\log P(x_i), $$ where $P$ means probability. This way, the Shannon entropy can be seen as the "average" or "expected" ...
0
votes
1answer
27 views

Entropy of $Y=bX$

If I have two random variables $Y$ and $U$ related as $Y=bU$, where $b>0$ is a constant and knowing that $\text{H}(x)$ represents the shannon entropy, such that: $$ \text{H}(x)=−\int \text{p}(x) ...
0
votes
0answers
28 views

Why is ln(p_i) not rounded down in theexpression for Shannon entropy?

Entropy supposedly " is the average amount of information contained in each message received"(Wikipedia: Entropy). However, to calculate the Shannon entropy for a finite sample, we have the sum over ...
1
vote
1answer
71 views

data processing inequality using non-deterministic functions

Generally data processing inequality says that the entropy cannot increase on applying a function f, or to be precise $H(f(X))\leq H(X)$ (also it is reversed if we know the function is k-to-1 so there ...
1
vote
1answer
78 views

Entropy of a uniform distribution

The entropy of a uniform distribution is $ ln(b-a)$. With $a=0$ and $b=1$ this reduces to zero. How come there is no uncertainty?
0
votes
0answers
17 views

Change of bases for entropy

From Cover and Thomas, Elements of Information Theory: Why isn't it: $ \log_b(p) = \frac{\log_a(p)}{\log_a(b)} $, so that $ H_a(X) $ is multiplied with $ \frac{1}{\log_b a} $?
2
votes
0answers
34 views

Finding the Legendre transform of an “entropy type” functional

I want to find the Legendre Transform of $$T(f) = \int_{\mathbb{R}^2} f \log \left(\frac{f}g{}\right) \, dx$$ on a set $H_M = \{ f: f \ge 0 \text{ and } \int_{\mathbb{R}^2} f =M\}$, where g is some ...
1
vote
0answers
87 views

Entropy calculation of Fibonacci distribution

For any positive integer $N$, consider the Fibonacci sequence $F_n$ of length $N$. Using $F_n$ we can define a Fibonacci discrete probability distribution as follows: $$p_N(n)=\frac{F_n}{\sum_{k=1}^N ...
1
vote
0answers
43 views

Estimate uncertainty of a function

I have a question about estimate the uncertainty of a function. I have a variable $x$. I assume that I can predict the variable $x$ is followed by a function $f(x)$ such as $$f(x)=\exp(-x^2)$$ Now, ...
7
votes
1answer
74 views

Structure of equientropic transformations

Given a probability vector $v=(v_1,\ldots,v_n)$ with $1\geq v_i\geq 0$ and $\sum_{i=1}^n v_i=1$ its entropy can be defined as: $$ H(v):=-\sum_{i=1}^nv_i\log v_i $$ I wonder what is known about ...
0
votes
0answers
16 views

A simpler way for an entropy inequality

I have to show that $\frac {1}{N}H(X_1,...,X_N)\le H(X_1)$. for a stationary stochastic process. I know that $H(X_1,...,X_N)=\Sigma _{i=1}^N H(X_i|X_1,...,X_{i-1})$. So far I have plugged that ...
0
votes
1answer
35 views

Mutual Information Entropy Inequality

I am trying to prove $H(x,y:z)>H(x:z)+H(y:z)$ and here is what I have. LHS: $=H(xy)-H(xy|z)=-\Sigma p(xy)lg(p(xy))+\Sigma p(xy|z)lg( \frac{p(xyz)}{p(z)})$ RHS: ...
0
votes
0answers
55 views

How to calculate the mutual information between two outputs of Rayleigh fading channels

We have the two channels: $$X_{a,i} = H_{i}s_{i} +N_{a,i} \\ X_{b,i} = H_{i}s_{i} +N_{b,i} $$ for $1 \leq i \leq n$, where $H_i$ denotes the i.i.d. channel coefficient and is a zero-mean complex ...
0
votes
1answer
26 views

Relative Entropy decomposition reference

may I ask for some reference pointers? My bad as I got a classic case of losing my reference and thus unsure what I wrote was right or wrong. I tried looking my old references and internet and didn't ...
0
votes
2answers
49 views

Proving certain aspects of Entropy

I am trying to prove three properties of entropy. $1)$ $H(X|Y,Z)\le H(X|Y)$ $2)$ $H(X|Y,Z)\le H(X,Y)$ $3)$ $H(X,Y,Z)+H(Y)\le H(X,Y)+H(Y,Z)$ I have proved the third one, but it is based on part 1. ...
0
votes
1answer
39 views

Motivation for the binary entropy function

What is the motivation for the definition of the binary entropy function $H(x) = -p\log_2(p) - (1-p)\log_2(1-p)$? I understand that we want the entropy to be zero at $p = 0$ and $p = 1$ (no ...
1
vote
1answer
98 views

Covering Number of Lipschitz Function Space

On page 12 of the slide deck here, the author gives an example where a lower bound (and upper bound, but I am particularly interested in the lower bound) on the $\epsilon$-covering number of a ...
2
votes
0answers
66 views

Limit of $\frac{1}{n}\log({n\choose np})$ without using Stirling's formula

I am trying to evaluate the following limit: $$ \forall p\in(0,1) ,\lim_{n\rightarrow \infty} \frac{1}{n}\log{n\choose \lfloor np \rfloor} =H(p),$$ where $\lfloor x\rfloor$ means the greatest integer ...
0
votes
0answers
51 views

Metric Entropy Upper Bounds

In the paper Information-Theoretic Determination of Minimax Rates of Convergence the authors present Theorem 3 as follows: If $M_2(\epsilon)$ is the $\ell_2$ packing entropy of a density class ...
0
votes
2answers
64 views

Question regarding the Entropy of a probability mass function

I assume that the entropy, $E$, of a probability mass function (pmf), $p(X)$, of a discrete random variable, $X$, is computed as: $$\begin{align}\mathbb{E}(p(X)) &= -p(X = x_1)\log[p(X = ...
2
votes
1answer
34 views

conditional entropies for identical distributions

Let me say I have two distributions $X$ and $Y$ which are identical, but they are not independent. Now if were to calculate the conditional entropies $H(X|Y)$ and $H(Y|X)$. Is calculating one joint ...
0
votes
0answers
76 views

Absolute value of difference between entropies (of two distributions)

I have the following inequality for the $L_1$ distance between two distributions $Q$, $Q^n$ on a finite set $B$: $$\|Q-Q^n|| < \frac{2|B|}{n}\leq \frac{C}{n} \leq \frac12 $$ Assuming $C\geq2|B|$, ...
2
votes
0answers
87 views

Shannon Entropy, prove $H(Wx)=H(x)+\log|\det W|$

I'm doing an essay on ICA (independent component analysis), and I could use some help. In essence, ICA is an algorithm that minimizes the entropy of $n$ $1$-dimensional random variables, but to show ...
1
vote
0answers
40 views

Entropy proerty

Let $a,b,c>0$ be distinct postive reals. Define four different probability distributions: $$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ ...
1
vote
1answer
60 views

Understanding an application of Entropy

I'm struggling with the following exercise on entropy. Suppose that your friend Alice chooses a number $X$ uniformly at random from $[1,n]$, which she writes down using $\log n$ bits; you can assume ...
0
votes
1answer
57 views

Inverse of shannon entropy

The shannon entropy of a bit $(p,1-p)$ is $$H(p)=-p\log(p)-(1-p)\log(1-p)$$. This is a well behaved function that uniquley assigns each state (up to permutation of its elements, i.e. ...
1
vote
1answer
136 views

Entropy of floating number array

I am familiar with shanon's definition of entropy. $$ H(P) = - \sum_{i=1}^n p_i \cdot \log_2(\mathcal p_i) $$ I am today in the situation that I'd like to compute an entropy like function but for a ...
1
vote
1answer
48 views

Entropy Solution of $u_t+(u^2/2)_x=0$

Given the initial data $$ g(x)= \cases{ 1 & x< -1 \\ 0 & -1 < x< 0 \\ 2 & 0 < x< 1 \\ 0 & 1 < x \\ } $$ What is the entropy solution of $u_t+(u^2/2)_x=0$?
2
votes
2answers
76 views

Reference for entropy of the binomial distribution?

The Wikipedia page Binomial distribution says that the entropy of the Binomial(n,p) is $\frac{1}{2}\log_2\left(2\pi e n p (1-p)\right) + O\left(\frac{1}{n}\right)$. What is a reference (paper or ...
0
votes
0answers
61 views

Entropy of beta-expansion

We have the transformation $T: [0,1) \rightarrow [0,1)$ given by $Tx = \beta x \text{ mod } 1$ with $\beta = \frac{1+ \sqrt{5}}{2}$. Calculate the entropy $h_{\mu}(T)$ of $T$ wrt the invariant ...
3
votes
1answer
84 views

Shannon Entropy Continuity Constraint

I have the following problem: I want to find the probability density $p$ which maximizes the Shannon entropy \begin{equation} S := - \int_{x_b}^{x_c} dx ~ p(x) \log (p(x)) \end{equation} under the ...
0
votes
1answer
79 views

Shannon entropy on the simplex

The state of a trit $$\{p_1,p_2,p_3=1-p_1-p_2\}$$ can be represented as a triangular simplex. The centre of the simplex is the maximally mixed state $$m=\{\frac{1}{3},\frac{1}{3},\frac{1}{3}\}$$. And ...
0
votes
1answer
20 views

Greenberg-Hastings-Model: What kind of shift space is it?

I would like to read something about the entropy of the one-dimensional Greenberg-Hastings-Model - and I think maybe I can find something about that in the book "Symbolic Dynamics and Coding" - but I ...
0
votes
1answer
54 views

Is this formula a KL divergence?

As everyone knows KL divergence's formula is $KL(p||q) = \sum_{i=1}^{n}p(i)\log (p(i)/q(i))$. In the image, formula(9) is really calculate KL(X||($(UZ^TA^T)$)) , however i have no idea why there is ...
1
vote
0answers
27 views

Summation methods and entropy

I am aware of the theory of divergent series, but don't know much of it. If you have a text to recommend, I'd be glad to hear it. Suppose I have an infinite-dimensional probability vector $\mathbf{p} ...
1
vote
2answers
80 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & ...
2
votes
1answer
122 views

Is there a symmetric alternative to Kullback-Leibler divergence?

I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity ...
1
vote
2answers
65 views

Given a Markov chain $X \rightarrow Y \rightarrow Z$, why is $I(X;Y|Z) \leq I(X;Y)$?

A Markov chain $X \rightarrow Y \rightarrow Z$ is given, where $X,Y,Z$ are random variables characterized by the probability distribution $p(x,y,z) = p(x)p(y|x)p(z|y)$. It follows that $I(X;Y) \geq ...
0
votes
1answer
52 views

What is the exact meaning of $I(X;Y|Z)$ in Information Theory?

I am wondering: is the notation $I(X;Y|Z)$ used to denote the mutual information between probabilities of $X$ and $Y|Z$ or between $X|Z$ and $Y|Z$?
2
votes
1answer
92 views

Entropy determination regarding lossless data compression

Suppose I had many computer data files to compress losslessly and wanted to know what is the theoretical limit to each one as far as minimum filesize possible. How would a math person go about doing ...