This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
1answer
16 views

Motivation for the binary entropy function

What is the motivation for the definition of the binary entropy function $H(x) = -p\log_2(p) - (1-p)\log_2(1-p)$? I understand that we want the entropy to be zero at $p = 0$ and $p = 1$ (no ...
0
votes
0answers
5 views

Covering Number of Lipschitz Function Space

On page 12 of the slide deck here, the author gives an example where a lower bound (and upper bound, but I am particularly interested in the lower bound) on the $\epsilon$-covering number of a ...
1
vote
0answers
42 views

Limit of $\frac{1}{n}\log({n\choose np})$ without using Stirling's formula

I am trying to evaluate the following limit: $$ \forall p\in(0,1) ,\lim_{n\rightarrow \infty} \frac{1}{n}\log{n\choose \lfloor np \rfloor} =H(p),$$ where $\lfloor x\rfloor$ means the greatest integer ...
0
votes
0answers
16 views

Metric Entropy Upper Bounds

In the paper Information-Theoretic Determination of Minimax Rates of Convergence the authors present Theorem 3 as follows: If $M_2(\epsilon)$ is the $\ell_2$ packing entropy of a density class ...
0
votes
2answers
13 views

Question regarding the Entropy of a probability mass function

I assume that the entropy, $E$, of a probability mass function (pmf), $p(X)$, of a discrete random variable, $X$, is computed as: $$\begin{align}\mathbb{E}(p(X)) &= -p(X = x_1)\log[p(X = ...
2
votes
1answer
18 views

conditional entropies for identical distributions

Let me say I have two distributions $X$ and $Y$ which are identical, but they are not independent. Now if were to calculate the conditional entropies $H(X|Y)$ and $H(Y|X)$. Is calculating one joint ...
0
votes
0answers
50 views

Absolute value of difference between entropies (of two distributions)

I have the following inequality for the $L_1$ distance between two distributions $Q$, $Q^n$ on a finite set $B$: $$\|Q-Q^n|| < \frac{2|B|}{n}\leq \frac{C}{n} \leq \frac12 $$ Assuming $C\geq2|B|$, ...
2
votes
0answers
64 views

Shannon Entropy, prove $H(Wx)=H(x)+\log|\det W|$

I'm doing an essay on ICA (independent component analysis), and I could use some help. In essence, ICA is an algorithm that minimizes the entropy of $n$ $1$-dimensional random variables, but to show ...
0
votes
0answers
32 views

Do Redundancy and Shannon's Entropy correlate?

I wonder can anyone answer whether the Entropy and the Redundancy correlate. I am interested in cases where the distribution is over discrete categories (i.e., cases), but I think it should apply for ...
1
vote
0answers
28 views

Entropy proerty

Let $a,b,c>0$ be distinct postive reals. Define four different probability distributions: $$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ ...
0
votes
0answers
18 views

equation for entropy of angles around a circle?

Hope you can help.. I'm using an random (even) distribution of angles to represent agents converging on a target value. (e.g zero degrees, where agents are at 4, 58, 99 & 202 degrees) I need to ...
1
vote
1answer
34 views

Understanding an application of Entropy

I'm struggling with the following exercise on entropy. Suppose that your friend Alice chooses a number $X$ uniformly at random from $[1,n]$, which she writes down using $\log n$ bits; you can assume ...
0
votes
1answer
23 views

Inverse of shannon entropy

The shannon entropy of a bit $(p,1-p)$ is $$H(p)=-p\log(p)-(1-p)\log(1-p)$$. This is a well behaved function that uniquley assigns each state (up to permutation of its elements, i.e. ...
2
votes
1answer
30 views

Entropy of floating number array

I am familiar with shanon's definition of entropy. $$ H(P) = - \sum_{i=1}^n p_i \cdot \log_2(\mathcal p_i) $$ I am today in the situation that I'd like to compute an entropy like function but for a ...
1
vote
1answer
27 views

Entropy Solution of $u_t+(u^2/2)_x=0$

Given the initial data $$ g(x)= \cases{ 1 & x< -1 \\ 0 & -1 < x< 0 \\ 2 & 0 < x< 1 \\ 0 & 1 < x \\ } $$ What is the entropy solution of $u_t+(u^2/2)_x=0$?
2
votes
1answer
31 views

Reference for entropy of the binomial distribution?

The Wikipedia page Binomial distribution says that the entropy of the Binomial(n,p) is $\frac{1}{2}\log_2\left(2\pi e n p (1-p)\right) + O\left(\frac{1}{n}\right)$. What is a reference (paper or ...
0
votes
0answers
46 views

Entropy of beta-expansion

We have the transformation $T: [0,1) \rightarrow [0,1)$ given by $Tx = \beta x \text{ mod } 1$ with $\beta = \frac{1+ \sqrt{5}}{2}$. Calculate the entropy $h_{\mu}(T)$ of $T$ wrt the invariant ...
2
votes
0answers
39 views

Shannon Entropy Continuity Constraint

I have the following problem: I want to find the probability density $p$ which maximizes the Shannon entropy \begin{equation} S := - \int_{x_b}^{x_c} dx ~ p(x) \log (p(x)) \end{equation} under the ...
0
votes
0answers
18 views

Shannon entropy on the simplex

The state of a trit $$\{p_1,p_2,p_3=1-p_1-p_2\}$$ can be represented as a triangular simplex. The centre of the simplex is the maximally mixed state $$m=\{\frac{1}{3},\frac{1}{3},\frac{1}{3}\}$$. And ...
0
votes
1answer
12 views

Greenberg-Hastings-Model: What kind of shift space is it?

I would like to read something about the entropy of the one-dimensional Greenberg-Hastings-Model - and I think maybe I can find something about that in the book "Symbolic Dynamics and Coding" - but I ...
0
votes
1answer
25 views

Is this formula a KL divergence?

As everyone knows KL divergence's formula is $KL(p||q) = \sum_{i=1}^{n}p(i)\log (p(i)/q(i))$. In the image, formula(9) is really calculate KL(X||($(UZ^TA^T)$)) , however i have no idea why there is ...
1
vote
0answers
17 views

Summation methods and entropy

I am aware of the theory of divergent series, but don't know much of it. If you have a text to recommend, I'd be glad to hear it. Suppose I have an infinite-dimensional probability vector $\mathbf{p} ...
0
votes
2answers
29 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & ...
1
vote
0answers
20 views

Is there a symmetric alternative to Kullback-Leibler divergence?

I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity ...
1
vote
1answer
33 views

Given a Markov chain $X \rightarrow Y \rightarrow Z$, why is $I(X;Y|Z) \leq I(X;Y)$?

A Markov chain $X \rightarrow Y \rightarrow Z$ is given, where $X,Y,Z$ are random variables characterized by the probability distribution $p(x,y,z) = p(x)p(y|x)p(z|y)$. It follows that $I(X;Y) \geq ...
0
votes
1answer
40 views

What is the exact meaning of $I(X;Y|Z)$ in Information Theory?

I am wondering: is the notation $I(X;Y|Z)$ used to denote the mutual information between probabilities of $X$ and $Y|Z$ or between $X|Z$ and $Y|Z$?
2
votes
1answer
36 views

Entropy determination regarding lossless data compression

Suppose I had many computer data files to compress losslessly and wanted to know what is the theoretical limit to each one as far as minimum filesize possible. How would a math person go about doing ...
1
vote
0answers
19 views

Entropy of Sum vs Difference of Random Variable

I am looking for a proof of the following fact Let X and X' be i.i.d on {0,1,2}(not necessarily uniform). Prove that $H(X + X' mod\;3) \leq H(X - X' mod\;3)$ where $H()$ is the standard Shannon ...
1
vote
2answers
46 views

If $g$ is a function of the random variable $X$, is it true that $H(X) = H(X) + H(g(X)\mid X)$?

I think my homework about entropy is formulated incorrectly. The question is the following: let $X$ be a discrete random variable. Show that the entropy of a function $g$ of $X$ is less than or ...
1
vote
0answers
39 views

Calculate $\lim_{n\rightarrow \infty}\frac{\log{\binom{n}{n_1}}}{n}$

We know that $\lim_{n\rightarrow \infty}\frac{n_1}{n}=p$ and $0\leq p\leq 1$. Based on this information I want to calculate $\lim_{n\rightarrow \infty}\frac{\log{\binom{n}{n_1}}}{n}$. Any help? Note: ...
3
votes
3answers
111 views

Entropy of convolution of measures

Let $G$ be a countable, discrete group, and let $\mu_1,\mu_2$ be probability measures on the group $G$. We define the entropy of $\mu_i$ as $H(\mu_i)=\sum\limits_{g \in G}-\mu_i(g)\log(\mu_i(g))$ ...
1
vote
1answer
38 views

Renyi entropy (zeroth order)

I am reading a book on information theory, therein has been introduced Renyi entropy of order $\alpha$ as $S_{\alpha} = \frac{1}{1-\alpha}\log(Tr\rho^{\alpha})$, where $\rho$ is density matrix. It ...
1
vote
0answers
38 views

Relative Entropy and variation formula for $C_c$

Let $R(\mu \mid \nu ) = \int_{\mathbb R} \log \frac{d\mu}{d\nu} d\nu$ for $\mu, \nu$ probability measures over $\mathbb R$. By the varational representation formula of Donsker and Varadhan we know ...
0
votes
1answer
55 views

Proving that $S=\bigcup_{j=0}^{2^k-1} S_{n-1+k}$ is a spanning set for the $2$-D Baker map

A set $S \subset X$ is a $(n,\epsilon)$-spanning set if $\forall x \in X$, $\exists y \in S $ such that $d_n(x,y)<\epsilon$. This is where we define $d_n(x,y)$ by $d_n(x,y)=\max_{0\leq k < ...
0
votes
0answers
18 views

Decomposing factorized entropy

I am trying to figure out how the equation for factorized entropy below is derived. The equation for entropy is $H(Q) = -\sum_x Q(x)\log Q(x)$ where $Q$ is a probability distribution. Let $Q(x) = ...
1
vote
2answers
48 views

Statistical Inference, Differential Geometry and Entropy

Context: Statistical Inference and Differential Geometry Let's consider a generic $ p(x;\theta) $ distribution with $ \theta $ Parameters Vector, it is obvious that $$ \int p(x; \theta) dx = 1 $$ ...
0
votes
1answer
73 views

Conditional Entropy in rolling a dice

A 6-sided die is tossed once. Two events X and Y are defined. X is the event in which an even number comes up and Y is the event in which the number is a multiple of 3. The value of H(X|Y) needs to be ...
1
vote
1answer
40 views

Channel Capacity of a Cycle Graph

I have the following problem: Given a discrete memoryless channel $Y = X + Z \mod5$, where $X$ is selected from one of 5 symbols (0, 1, 2, 3, 4), $Z$ randomly selected from (-1, 0, 1), and $X$ and $Z$ ...
0
votes
1answer
32 views

prove a theorem about an upper bound of entropy of a random vector

There is a theorem that: if Z is any zero-mean, complex random vector with covariance $E[ZZ^H]=R_z$, then $H(Z)\leq \log|{\pi eR_z}|$, with equality holding if and only if Z has a circularly ...
0
votes
0answers
16 views

Entropy of bit position in a bit stream

8 bit strings are sent over a channel. First two bits are always 1. Last six bits can be either 0 or 1. Receiver randomly selects bit-position and reveals bit but not its position. If X is the random ...
3
votes
2answers
59 views

Proof of recursivity of Shannon's Entropy

Does anybody know a book where the proof of recursivity property of Shannon's Entropy can be found? I mean this: $$H(q_1,...,q_n)=H(q_1 + q_2, q_3,...,q_n) + (q_1 +q_2)H( \frac{q_1}{q_1+q_2} , ...
0
votes
1answer
55 views

Calculate the Entropy Change if 5 Previously Tossed Dice Are Turned to All “1”

Relevant Equations: S = Boltzmann*ln(W) where S is entropy and W is the number of microstates. I have thought about this two ways. 1 way. Look at each die separately. Let macrostate 1 = number of ...
1
vote
1answer
107 views

Prove there exist a $p$ so that the inequality holds

I am stuck with the following problem. Given the Gaussian mixture distribution $f(\cdot)$ $$ f(x) = ...
0
votes
0answers
59 views

How to compute the topological entropy of a permutation?

I have a permutation, say as ${4,1,7,2,3,5,6}$, given by its induced matrix. According to this paper (Proposition 11 on p. 82), To compute its topological entropy, one can compute the ...
2
votes
1answer
85 views

Relationship between compression, shannon entropy and kolmogorov complexity

I have read that the Shannon Entropy is used as a bound for the compressibility of a message, for example here 1 it says "In other words, the best possible lossless compression rate is the entropy ...
0
votes
1answer
70 views

prove this inequality related to probability and information theory

How do I prove this? I'm thinking I should use Jensen's inequality somehow. $$\sum_K p_k(1-p_k) \le -\sum_K p_k\log p_k$$ The assumption that $\sum_K p_k=1$ holds.
0
votes
1answer
38 views

Normalization of data in decision tree

After reading through a few references, I have come to know that for machine learning in general, it is necessary to normalize features so that no features are arbitrarily large ($centering$) and all ...
0
votes
1answer
24 views

Tools to compress a finite list as a function

Can someone show me some tool to a lossless compression in an algorithm of a finite list of rational numbers? By example this list A=(0,1,3,2,-1,-2,0), there is a way to construct an algorithm or ...
1
vote
1answer
52 views

What is the correct equation for conditional relative entropy and why

I was trying to understand the concept of conditional relative entropy. As in: $$D(P(X\mid Y) ||Q(X\mid Y))= E [\log\frac{P(X\mid Y)}{Q(X\mid Y)}]$$ I would have thought that its equations would ...
5
votes
1answer
79 views

Counting graph isomorphisms and entropy

Question: If all graphs on $n$ vertices are given equal probability, what does the induced probability distribution on the graph isomorphism classes look like? Are there any patterns that emerge as ...