This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
0answers
8 views

Entropy proerty

Let $a,b,c>0$ be distinct postive reals. Define four different probability distributions: $$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ ...
0
votes
0answers
11 views

equation for entropy of angles around a circle?

Hope you can help.. I'm using an random (even) distribution of angles to represent agents converging on a target value. (e.g zero degrees, where agents are at 4, 58, 99 & 202 degrees) I need to ...
1
vote
1answer
27 views

Understanding an application of Entropy

I'm struggling with the following exercise on entropy. Suppose that your friend Alice chooses a number $X$ uniformly at random from $[1,n]$, which she writes down using $\log n$ bits; you can assume ...
0
votes
1answer
20 views

Inverse of shannon entropy

The shannon entropy of a bit $(p,1-p)$ is $$H(p)=-p\log(p)-(1-p)\log(1-p)$$. This is a well behaved function that uniquley assigns each state (up to permutation of its elements, i.e. ...
2
votes
1answer
24 views

Entropy of floating number array

I am familiar with shanon's definition of entropy. $$ H(P) = - \sum_{i=1}^n p_i \cdot \log_2(\mathcal p_i) $$ I am today in the situation that I'd like to compute an entropy like function but for a ...
1
vote
1answer
24 views

Entropy Solution of $u_t+(u^2/2)_x=0$

Given the initial data $$ g(x)= \cases{ 1 & x< -1 \\ 0 & -1 < x< 0 \\ 2 & 0 < x< 1 \\ 0 & 1 < x \\ } $$ What is the entropy solution of $u_t+(u^2/2)_x=0$?
2
votes
1answer
27 views

Reference for entropy of the binomial distribution?

The Wikipedia page Binomial distribution says that the entropy of the Binomial(n,p) is $\frac{1}{2}\log_2\left(2\pi e n p (1-p)\right) + O\left(\frac{1}{n}\right)$. What is a reference (paper or ...
0
votes
0answers
42 views

Entropy of beta-expansion

We have the transformation $T: [0,1) \rightarrow [0,1)$ given by $Tx = \beta x \text{ mod } 1$ with $\beta = \frac{1+ \sqrt{5}}{2}$. Calculate the entropy $h_{\mu}(T)$ of $T$ wrt the invariant ...
2
votes
0answers
38 views

Shannon Entropy Continuity Constraint

I have the following problem: I want to find the probability density $p$ which maximizes the Shannon entropy \begin{equation} S := - \int_{x_b}^{x_c} dx ~ p(x) \log (p(x)) \end{equation} under the ...
0
votes
0answers
14 views

Shannon entropy on the simplex

The state of a trit $$\{p_1,p_2,p_3=1-p_1-p_2\}$$ can be represented as a triangular simplex. The centre of the simplex is the maximally mixed state $$m=\{\frac{1}{3},\frac{1}{3},\frac{1}{3}\}$$. And ...
0
votes
1answer
9 views

Greenberg-Hastings-Model: What kind of shift space is it?

I would like to read something about the entropy of the one-dimensional Greenberg-Hastings-Model - and I think maybe I can find something about that in the book "Symbolic Dynamics and Coding" - but I ...
0
votes
1answer
24 views

Is this formula a KL divergence?

As everyone knows KL divergence's formula is $KL(p||q) = \sum_{i=1}^{n}p(i)\log (p(i)/q(i))$. In the image, formula(9) is really calculate KL(X||($(UZ^TA^T)$)) , however i have no idea why there is ...
0
votes
0answers
14 views

Summation methods and entropy

I am aware of the theory of divergent series, but don't know much of it. If you have a text to recommend, I'd be glad to hear it. Suppose I have an infinite-dimensional probability vector $\mathbf{p} ...
0
votes
2answers
26 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & ...
1
vote
0answers
17 views

Is there a symmetric alternative to Kullback-Leibler divergence?

I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity ...
1
vote
1answer
33 views

Given a Markov chain $X \rightarrow Y \rightarrow Z$, why is $I(X;Y|Z) \leq I(X;Y)$?

A Markov chain $X \rightarrow Y \rightarrow Z$ is given, where $X,Y,Z$ are random variables characterized by the probability distribution $p(x,y,z) = p(x)p(y|x)p(z|y)$. It follows that $I(X;Y) \geq ...
0
votes
1answer
40 views

What is the exact meaning of $I(X;Y|Z)$ in Information Theory?

I am wondering: is the notation $I(X;Y|Z)$ used to denote the mutual information between probabilities of $X$ and $Y|Z$ or between $X|Z$ and $Y|Z$?
2
votes
1answer
25 views

Entropy determination regarding lossless data compression

Suppose I had many computer data files to compress losslessly and wanted to know what is the theoretical limit to each one as far as minimum filesize possible. How would a math person go about doing ...
1
vote
0answers
17 views

Entropy of Sum vs Difference of Random Variable

I am looking for a proof of the following fact Let X and X' be i.i.d on {0,1,2}(not necessarily uniform). Prove that $H(X + X' mod\;3) \leq H(X - X' mod\;3)$ where $H()$ is the standard Shannon ...
1
vote
2answers
45 views

If $g$ is a function of the random variable $X$, is it true that $H(X) = H(X) + H(g(X)\mid X)$?

I think my homework about entropy is formulated incorrectly. The question is the following: let $X$ be a discrete random variable. Show that the entropy of a function $g$ of $X$ is less than or ...
1
vote
0answers
38 views

Calculate $\lim_{n\rightarrow \infty}\frac{\log{\binom{n}{n_1}}}{n}$

We know that $\lim_{n\rightarrow \infty}\frac{n_1}{n}=p$ and $0\leq p\leq 1$. Based on this information I want to calculate $\lim_{n\rightarrow \infty}\frac{\log{\binom{n}{n_1}}}{n}$. Any help? Note: ...
3
votes
3answers
106 views

Entropy of convolution of measures

Let $G$ be a countable, discrete group, and let $\mu_1,\mu_2$ be probability measures on the group $G$. We define the entropy of $\mu_i$ as $H(\mu_i)=\sum\limits_{g \in G}-\mu_i(g)\log(\mu_i(g))$ ...
1
vote
1answer
36 views

Renyi entropy (zeroth order)

I am reading a book on information theory, therein has been introduced Renyi entropy of order $\alpha$ as $S_{\alpha} = \frac{1}{1-\alpha}\log(Tr\rho^{\alpha})$, where $\rho$ is density matrix. It ...
1
vote
0answers
29 views

Relative Entropy and variation formula for $C_c$

Let $R(\mu \mid \nu ) = \int_{\mathbb R} \log \frac{d\mu}{d\nu} d\nu$ for $\mu, \nu$ probability measures over $\mathbb R$. By the varational representation formula of Donsker and Varadhan we know ...
0
votes
1answer
53 views

Proving that $S=\bigcup_{j=0}^{2^k-1} S_{n-1+k}$ is a spanning set for the $2$-D Baker map

A set $S \subset X$ is a $(n,\epsilon)$-spanning set if $\forall x \in X$, $\exists y \in S $ such that $d_n(x,y)<\epsilon$. This is where we define $d_n(x,y)$ by $d_n(x,y)=\max_{0\leq k < ...
0
votes
0answers
17 views

Decomposing factorized entropy

I am trying to figure out how the equation for factorized entropy below is derived. The equation for entropy is $H(Q) = -\sum_x Q(x)\log Q(x)$ where $Q$ is a probability distribution. Let $Q(x) = ...
1
vote
2answers
45 views

Statistical Inference, Differential Geometry and Entropy

Context: Statistical Inference and Differential Geometry Let's consider a generic $ p(x;\theta) $ distribution with $ \theta $ Parameters Vector, it is obvious that $$ \int p(x; \theta) dx = 1 $$ ...
0
votes
1answer
63 views

Conditional Entropy in rolling a dice

A 6-sided die is tossed once. Two events X and Y are defined. X is the event in which an even number comes up and Y is the event in which the number is a multiple of 3. The value of H(X|Y) needs to be ...
1
vote
1answer
33 views

Channel Capacity of a Cycle Graph

I have the following problem: Given a discrete memoryless channel $Y = X + Z \mod5$, where $X$ is selected from one of 5 symbols (0, 1, 2, 3, 4), $Z$ randomly selected from (-1, 0, 1), and $X$ and $Z$ ...
0
votes
1answer
27 views

prove a theorem about an upper bound of entropy of a random vector

There is a theorem that: if Z is any zero-mean, complex random vector with covariance $E[ZZ^H]=R_z$, then $H(Z)\leq \log|{\pi eR_z}|$, with equality holding if and only if Z has a circularly ...
0
votes
0answers
15 views

Entropy of bit position in a bit stream

8 bit strings are sent over a channel. First two bits are always 1. Last six bits can be either 0 or 1. Receiver randomly selects bit-position and reveals bit but not its position. If X is the random ...
3
votes
2answers
55 views

Proof of recursivity of Shannon's Entropy

Does anybody know a book where the proof of recursivity property of Shannon's Entropy can be found? I mean this: $$H(q_1,...,q_n)=H(q_1 + q_2, q_3,...,q_n) + (q_1 +q_2)H( \frac{q_1}{q_1+q_2} , ...
0
votes
1answer
53 views

Calculate the Entropy Change if 5 Previously Tossed Dice Are Turned to All “1”

Relevant Equations: S = Boltzmann*ln(W) where S is entropy and W is the number of microstates. I have thought about this two ways. 1 way. Look at each die separately. Let macrostate 1 = number of ...
1
vote
1answer
106 views

Prove there exist a $p$ so that the inequality holds

I am stuck with the following problem. Given the Gaussian mixture distribution $f(\cdot)$ $$ f(x) = ...
0
votes
0answers
56 views

How to compute the topological entropy of a permutation?

I have a permutation, say as ${4,1,7,2,3,5,6}$, given by its induced matrix. According to this paper (Proposition 11 on p. 82), To compute its topological entropy, one can compute the ...
2
votes
1answer
74 views

Relationship between compression, shannon entropy and kolmogorov complexity

I have read that the Shannon Entropy is used as a bound for the compressibility of a message, for example here 1 it says "In other words, the best possible lossless compression rate is the entropy ...
0
votes
1answer
68 views

prove this inequality related to probability and information theory

How do I prove this? I'm thinking I should use Jensen's inequality somehow. $$\sum_K p_k(1-p_k) \le -\sum_K p_k\log p_k$$ The assumption that $\sum_K p_k=1$ holds.
0
votes
1answer
36 views

Normalization of data in decision tree

After reading through a few references, I have come to know that for machine learning in general, it is necessary to normalize features so that no features are arbitrarily large ($centering$) and all ...
0
votes
1answer
23 views

Tools to compress a finite list as a function

Can someone show me some tool to a lossless compression in an algorithm of a finite list of rational numbers? By example this list A=(0,1,3,2,-1,-2,0), there is a way to construct an algorithm or ...
1
vote
1answer
49 views

What is the correct equation for conditional relative entropy and why

I was trying to understand the concept of conditional relative entropy. As in: $$D(P(X\mid Y) ||Q(X\mid Y))= E [\log\frac{P(X\mid Y)}{Q(X\mid Y)}]$$ I would have thought that its equations would ...
5
votes
1answer
70 views

Counting graph isomorphisms and entropy

Question: If all graphs on $n$ vertices are given equal probability, what does the induced probability distribution on the graph isomorphism classes look like? Are there any patterns that emerge as ...
0
votes
0answers
45 views

Entropy derivation from Multiplicity

Multiplicity(W)= N!/(n1!*n2!....ni!) Entropy = 1/N * ln W = 1/N*ln N! - 1/N*sigma_for_all_i(ln ni!) As N->infinity, By Stirlings approximation ...
0
votes
1answer
25 views

Meaning of this term:$H(X \oplus\hat{X}|\hat{X} )$

Here, $H$ means the entropy function. I understand that the symbol $\oplus$ means modulo $2$ addition. But I don't understand the significance of the entire expression.
1
vote
1answer
31 views

Relation between entropy and compressibility of a file

Suppose I have an ordered list of bytes (the hexdump of some object file), and wish to calculate the information entropy of this file. My understanding is I can calculate this as $$ \sum_{n=0}^{n=255} ...
1
vote
2answers
154 views

Upper bound on the entropy of a sum two random variables

Let $X$ be a random variable such that $|X| \leq A$ almost surely, for some $A > 0$. Let $Z$ be independent of $X$ such that $Z \sim {\cal N}(0, N)$. My question is: How large can the entropy ...
1
vote
2answers
91 views

Measuring the entropy of a graph representing a transition probability matrix of a first order markov chain

There's a research project i'm currently working on which requires me to analyze various aspects of "worlds" represented by transition probability matrices, where the nodes represent objects in the ...
0
votes
0answers
27 views

Finite shannon entropy and mutual information

I was wondering, if it can be shown that the shannon entropy for continuous random variables are finite, such that $H(X)=-\int_{-\infty}^{\infty}f(x)*log(f(x)) dx < \infty$ and the same question ...
2
votes
1answer
206 views

Entropy of the product of two random variables

Consider a random matrix $X$ and a random vector $Y$. Let the Shannon entropies $H(X) = H(Y) = n$. Is there a simple upper bound for entropy $H(XY)$? I believe $H(XY) \leq 2n$ as that is a simple ...
2
votes
1answer
54 views

partition with infinite entropy?

Let $P$ be an infinite partition of the interval $[0,1]$. Let $P$ have elements $I_i$ which has Lebesgue measure $m(I_i)$. Then the entropy of $P$ is defined by $\sum_i -m(I_i)\log m(I_i)$. Can this ...
1
vote
0answers
17 views

Mutual information staying constant under composition of channels

Consider the following scenario: one has 2 communication channels $C_1$ and $C_2$. Let $p_0(x)$ be some arbitrary but fixed input probability distribution. The mutual information between the input ...