This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
1answer
120 views

Question about Mutual Information

I am learning about mutual information, and am confused about one of the definitions. Mutual information is defined as $ I(X;Y) = H(X) - H(X | Y) $ where, $$ H(X) = \sum_{x} p(x) \log ...
4
votes
1answer
125 views

Definition of entropy of an ergodic measure

I'm reading a paper in which it is stated that The entropy of an ergodic measure is defined as $$\lim_{n \to \infty} -\frac{1}{n} \sum_{|w|=n} \mu[w] \log \mu[w].\tag{1} \label{eq:1}$$ Here ...
1
vote
0answers
107 views

Entropy of a Finite State Transducer

Theorem 7 in Shannon's seminal paper A Mathematical Theory of Communication states: "The output of a finite state transducer driven by a finite state statistical source is a finite state ...
3
votes
1answer
161 views

Relative entropy between singular measures

Usually, to define relative entropy between two probability measures, one assumes absolute continuity. Is it possible to extend the usual definition in the non absolutely continuous case?
1
vote
1answer
87 views

Upper bound for $-t \log t$

While reading Csiszár & Körner's "Information Theory: Coding Theorems for Discrete Memoryless Systems", I came across the following argument: Since $f(t) \triangleq -t\log t$ is concave and ...
0
votes
1answer
100 views

Test for randomness

I'm trying to write a program to compute a metric for the entropy in files to determine a probability that the file is compressed or encrypted. Compressed and encrypted files have very, very, very ...
1
vote
0answers
79 views

Showing that Normalized Redundancy is nonreliant on the properties of Bijection and Monotonicity

In information theory, the concept of mutual information states that for two features of arbitrary discretized probability, the following formula holds true: \begin{aligned} I(X;Y) = \sum_{y \in Y} ...
1
vote
0answers
139 views

Inequality involving KL divergence

Following is a part of an answer which was not resolved when I tried to answer a question in mathoverflow. I thought it would be nice to discuss that here. Let $P$ and $Q$ be two distinct ...
3
votes
1answer
325 views

Does a maximum entropy probability distribution with KL-divergence constraint not exist?

In my earlier question I asked about a technical aspect of solving a system of equations arising from looking for an entropy-maximizing distribution $p(x)$ continuous on $\mathbb{R}$ and constrained ...
3
votes
0answers
246 views

Can I map entropy values to a common range so that they are comparable?

I am using the standard Shannon entropy formula for calculating the entropy of a system at different states. The system has a different number of possible outcomes at each state, in other words the ...
4
votes
2answers
890 views

Can I normalize KL-divergence to be $\leq 1$?

The Kullback-Leibler divergence has a strong relationship with mutual information, and mutual information has a number of normalized variants. Is there some similar, entropy-like value that I can use ...
1
vote
1answer
226 views

How do I go about calculating the entropy level of this algorithm?

I have a set of items. These items are (pseudo)randomly placed into buckets. The buckets are ordered and items placed in them are ordered. After all of the items are placed in buckets, the items ...
3
votes
1answer
255 views

Entropy of $X =\{1,2,\ldots,\infty\}$ with the probability of $\{1/2^1,1/2^2,\ldots,1/2^\infty\}$?

I'm studing for an information theory exam, maybe some of you can help me here with an exercise. What's the entropy of $X$ as $\{1,2,\ldots,n\}$ ($n$=infinity) where the probabilities are $P \{1/2^1, ...
22
votes
2answers
1k views

An information theory inequality which relates to Shannon Entropy

For $a_1,...,a_n,b_1,...,b_n>0,\quad$ define $a:=\sum a_i,\ b:=\sum b_i,\ s:=\sum \sqrt{a_ib_i}$. Is the following inequality true?: $${\frac{\Bigl(\prod a_i^{a_i}\Bigr)^\frac1a}a \cdot ...
0
votes
1answer
240 views

Relative Entropy given two non-equivalent sets

I am trying to calculate the relative entropy given two collections and have a question regarding some issues. Supposed we have two sets, $Real$ and $Calculated$, and their respective probability ...
4
votes
2answers
1k views

Derivation of the maximum entropy distribution

I am reading a book and having trouble following something. The problem is to try to maximize the differential entropy $-\int_{0}^{\infty}p(r)\log p(r)$ under the constraints that ...