This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

3
votes
1answer
42 views

Definition of topological entropy

What the meaning of the limit that appears in the definition of topological entropy? Let $X$ a compact metric space and $f\colon X\to X$ a continuous function and a subset $K \subset X$. The ...
0
votes
0answers
16 views

a problem involving binary entropy function

let $\alpha<1/2$ such that $2^{H(\alpha)}\le 2^{1-\epsilon}$,when $H$ is binary entropy function. how can i prove that then we have: $2^{n(1-\epsilon)}\ge \sum\limits_{i\le \alpha n } {n \choose ...
0
votes
2answers
41 views

Partition-based entropy of a sequence

The entropy $H$ of a discrete random variable $X$ is defined by $$H(X)=E[I(X)]=\sum_xP(x)I(x)=\sum_xP(x)\log P(x)^{-1}$$ where $x$ are the possible values of $X$, $P(x)$ is the probability of $x$, ...
-3
votes
0answers
16 views

what is the mathematical semantic of information (in the context of entropy theory)? [closed]

can you discuss the semantic of this definition. "information is change in the entropy” can you validate this mathematically and if possible refer this definition to any relevant literature
1
vote
1answer
31 views

Calculating Shannon Entropy for DNA sequence?

I'm following the formula on http://www.shannonentropy.netmark.pl/calculate to calculate the Shannon Entropy of a string of nucleotides [nt]. Since their are 4 nt, ...
2
votes
2answers
47 views

$\log (A + \delta A) = ?$ (as an expansion in $\delta A$), where $A$ and $\delta A $ are matrices

$A$ and $\delta A$ are two non-commuting matrices and I am seeking a power series expansion to 2nd order in $\delta A$. After writing it as $\log (A (1 + A^{-1}\delta A) )$, I am unable to figure out ...
1
vote
2answers
52 views

Scalar derivative of ${\rm tr}~[A(x)\log A(x)]$ where $A(x)$ is a square matrix

How do i proceed to calculate $$\frac{d}{dx}{\rm tr}\left[{A(x) \log A(x)}\right]$$ where $A(x) \in \mathbb{M}(n)$ and $x \in \mathbb{R}$? The $\log$ function is the one defined by the exponential ...
0
votes
0answers
34 views

Zero conditional entropy

This question is related to this math.se question but I need a bit more guidance. For two discrete random variables $X,Y$ we define their conditional entropy to be $$H(X|Y) = -\sum_{y \in Y} Pr[Y = ...
0
votes
1answer
23 views

Is mutual information convex in the joint distribution?

Assume some joint distribution $P(X,Y) = P(Y|X)P(X)$. It is well know that, for fixed $P(Y|X)$, mutual information is a concave function of $P(X)$ and, for fixed $P(X)$, a convex function of $P(Y|X)$ ...
1
vote
2answers
37 views

Concentration property of entropy

Let $X$ be a random variable taking its values in $A = \{a_1,\ldots,a_n\}$ such that $Pr[X = a_i] = p_i$ for all $1 \leq i \leq n.$ The entropy of $X$ is defined as $$H(X) = -\sum_{i=1}^n p_i ...
4
votes
1answer
415 views

Notation of cross entropy

I have a question regarding a notation that seems to be very usual. For starters, cross entropy is defined by: \begin{align}H(X, q) &= H(X) + D(p||q) \\ & =-\sum_x p(x)\log_2 q(x)\end{align} ...
2
votes
2answers
52 views

Differential Entropy

I'm a little temporarily confused about the concept of differential entropy. It says on wikipedia that the differential entropy of a Gaussian is $\log(\sigma\sqrt{2\pi e})$. However I was thinking as ...
3
votes
1answer
69 views

“Empirical” entropy.

Information entropy is usually defined as $$\text{I}_b({\bf p}) = -\sum_{\forall i}p_i\log_b(p_i)$$ i.e. the expected value of the negative logarithm of the probabilities. This is all good when we ...
2
votes
1answer
86 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
1
vote
1answer
22 views

Shannon information for a set of 3 equally probable elements?

How to calculate entropy as a number of binary choices for a set of three equally probable elements? The Shannon's formula gives $\log_2(3)=1.585$. But any interpretation of binary choices gives me ...
1
vote
0answers
27 views

Is the Library of Babel random? Does it contain information?

The Library of Babel is defined as a universe in the form of a vast library containing all possible 410-page books of a certain format and character set. However, applying two means of ...
0
votes
0answers
15 views

Why $H_{V^* \cup W^*} > H_{V \cup W}$ if $H_V$ denotes entropy of language

Let $W \subseteq X^*$ be an infinite language over a finite alphabet $X$, and define ($|w|$ denotes the length of $w \in W$) $$ H_W := \limsup_{n\to \infty} \frac{\log_{|X|} | \{ w \in w \in W, |w| = ...
2
votes
1answer
45 views

Entropy of $f(x)=1$

Let $f(x)$ be a probability mass function $f(x) = 1$ on $x = [0,1]$, and entropy defined as $$H(p(x)) = -\int p(x) \log_2(p(x)) \, dx$$ where $p(x)$ is a pmf. Unless I've made an arithmetic error, the ...
1
vote
1answer
295 views

Is maximizing the Shannon differential entropy equivalent to minimizing the predictability and/or minimizing the maximum density?

For a real-valued, 1-dimensional, continuous random variable X with density f(x), I am trying to determine if maximizing the Shannon differential entropy of f(x) is mathematically equivalent to ...
0
votes
1answer
10 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
0
votes
0answers
36 views

Mutual Information: Are these two equations equal?

I'm working with Multivariate Mutual Information (MMI), specifically with three variables $(X,Y,Z)$, applied to RNA sequences. The MMI equation that I use for three variables is based on entropy ...
3
votes
1answer
199 views

Relative entropy between singular measures

Usually, to define relative entropy between two probability measures, one assumes absolute continuity. Is it possible to extend the usual definition in the non absolutely continuous case?
0
votes
1answer
46 views

Approximation for a binomial coefficient sequence summation

What is a good approximation to $$\dfrac{{\binom{k}{i}}{\binom{k}{i}}(i-1)!}{\binom{k(k-1)/2}{i}}$$ $$\dfrac{{\binom{k}{i}}{\binom{k}{i}}(i-1)!}{(2^{(\log ...
0
votes
1answer
27 views

How to calculate entropy from a set of samples?

entropy (information content) is defined as: $$ H(X) = \sum_{i} {\mathrm{P}(x_i)\,\mathrm{I}(x_i)} = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)} $$ This allows to calculate the entropy of a ...
0
votes
0answers
20 views

How to efficiently select a subset of elements that maximizes a certain property? (entropy)

I need to select $k$ elements from a pool containing a much larger number $N$ of elements. The selection must be done in a way that a function $h(\{z_{i_1},\ldots,z_{i_k}\})$ is maximized or ...
3
votes
1answer
52 views

Fano's Inequality Proof

For an information theory class, I am studying the proof for Fano's inequality, i.e.: $H(P_e) + P_elog(|X|) \geq H(X|\hat{X}) \geq H(X|Y)$ Where $H(X)$ is the entropy of the random variable X ...
1
vote
1answer
46 views

Is conditional entropy ever taken to be a random variable?

In probability theory, the conditional expectation $E(X|Y)$ and variance $V(X|Y)$ er usually taken to be random variables, st. the value of $E(X|Y)$ depends on what value $Y$ ends up taking. I've ...
2
votes
0answers
39 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
1
vote
1answer
23 views

How does a wavelet help in compressing data

I have an understanding of how we carry out image compression by using DCT along with Huffman encoding. The next subject is wavelets. I understand that wavelets are small waves and there are different ...
0
votes
1answer
52 views

Shannon entropy on the simplex

The state of a trit $$\{p_1,p_2,p_3=1-p_1-p_2\}$$ can be represented as a triangular simplex. The centre of the simplex is the maximally mixed state $$m=\{\frac{1}{3},\frac{1}{3},\frac{1}{3}\}$$. And ...
0
votes
0answers
51 views

Shannon Entropy Maximization with Constraints

I have got a cumulative distribution function $F_X(x)=Pr(X<=x)$. This distribution is described by 2 parameters $\alpha, \beta$. We define $F_k$ as follows: $\forall k<=n_k, ...
1
vote
2answers
59 views

Given a Markov chain $X \rightarrow Y \rightarrow Z$, why is $I(X;Y|Z) \leq I(X;Y)$?

A Markov chain $X \rightarrow Y \rightarrow Z$ is given, where $X,Y,Z$ are random variables characterized by the probability distribution $p(x,y,z) = p(x)p(y|x)p(z|y)$. It follows that $I(X;Y) \geq ...
0
votes
1answer
36 views

Clarifying Derivation of Entropy

I'm learning about probability from the book Pattern Recognition and Machine Learning by Christopher Bishop. It includes a justification for the definition of entropy that can be summarized as: let ...
0
votes
1answer
30 views

Mutual Information: How these two equations are equal?

I'm a biologist trying to apply the Mutual Information (MI) to some RNA secondary structure. I know that there exists two MI equation that, mathematically, are equal: $I(X,Y) = \sum_{x,y} p(x,y) ...
0
votes
0answers
25 views

The difference in entropy rates between a hidden process and its observation

Let $S$ be a finite state space and $o:S\to S$ an observation function. Given a distribution $p$ on $S\times S$, consider the following optimization problem: $$\max \left[ EntropyRate(\{x_t\}) - ...
1
vote
1answer
31 views

Infinite closed shift-invariant set of binary sequences with zero entropy?

Does there exist an infinite closed shift-invariant $X \subset \{0,1\}^\Bbb Z$ with zero topological entropy? How to think of an example? Will periodic points of shift$|_X$ have zero entropy?
0
votes
1answer
37 views

Derivation of equation for self information

I am trying to understand how the formula I(x) = -log(p(x)) for self information was derived. From what I have read, 2 constraints were imposed on the properties ...
9
votes
3answers
379 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a smooth function defined on $\textbf{R}^d$. What are the assumptions I should use to assume that $$\operatorname{div}\left(\nabla G(x) +xG(x)\right)=0 \quad (\forall x\in \textbf{R}^d)$$ ...
2
votes
0answers
15 views

Graph Entropy - A Tractable Measure to Measure Distinguishability of Neighbourhoods

Given a labelled directed graph G, I am interested in a measurement of G that captures how distinguishable arbitrary connected sub graphs of G are. Labels may repeat and as such two or more different ...
0
votes
0answers
12 views

How to define a one-parameter family of probability distributions

I am trying to evaluate a noise-source as a means of providing entropy to a random number generator. I am running into trouble when it comes to determining the probability distribution that has the ...
1
vote
2answers
42 views

How much information does learning this interval give you?

Let's say you have a number $x$, and a priori, you know that $x \in [0, 1)$ (each value from 0 to 1 is equally likely.) Then a wizard comes and tells you that $x \in [a, b) \subseteq [0, 1)$. How much ...
0
votes
1answer
63 views

mutual information adds along path

Is it true that $I(X;Y)+I(Y;Z)=I(X;Z)$ for $X \to Y \to Z$? $I(X;Z) = H(X)+H(Z)-H(X,Z)$ and $I(X;Y)+I(Y;Z) = H(X)+H(Z)-H(Z|Y)-H(X|Y)$ Hence, we would require $-H(X,Z)=-H(Z|Y)-H(X|Y)$ -- is it true? ...
1
vote
2answers
66 views

How many bit-flips are required to achieve random distribution?

If I have a binary number W bits wide, initially all set to zero, and I repeatedly pick a random bit and toggle it from zero to one or vice versa, how many times would I need to do this to achieve ...
0
votes
1answer
14 views

Matrix entropy measure

I have a matrix (its dimension is $n$ x $m$) where each cell can be $0$ or $1$. I would like to calculate an "entropy" measure on it that tells me how close are the ones together or how spread they ...
1
vote
2answers
59 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & ...
-1
votes
2answers
33 views

Markov chain and conditional entropy [closed]

Markov chain (DTMC) is described by transition matrix: $$\textbf{P} = \begin{pmatrix}0 & 1\\ \frac{1}{2} & \frac{1}{2} \end{pmatrix}.$$ Initial distribution $X_1 = \left(\frac{1}{4}, ...
0
votes
1answer
45 views

For $A,B,C$ independent and normal, what is $I(A+B;\ A+C)$?

Say $A,B,C$ are mutually independent and normally distributed with zero mean but possibly different variances $\sigma_1,\sigma_2,\sigma_3$. What is the mutual information between $A+B$ and $A+C$? All ...
2
votes
1answer
68 views

Topological Entropy of $T$, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B\cup C,T_{|A\cup B\cup C})$ denote the toplogical entropy of $T$, restricted on $A\cup B\cup C$, where $A,B,C\subset X$ ...
1
vote
1answer
43 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove (or disprove) that $$ H(X) \leq ...
0
votes
0answers
12 views

Which argument in KL Divergence minimization?

The KL divergence $D_{KL}(p||q) = p^T\ln(\frac{p}{q})$ is not a distance measure because first of all it is not symmetric. In applications, one usually has a prior distribution, say $y$, and wants ...