This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

-3
votes
0answers
14 views

what is the mathematical semantic of information (in the context of entropy theory)? [on hold]

can you discuss the semantic of this definition. "information is change in the entropy” can you validate this mathematically and if possible refer this definition to any relevant literature
1
vote
1answer
29 views

Calculating Shannon Entropy for DNA sequence?

I'm following the formula on http://www.shannonentropy.netmark.pl/calculate to calculate the Shannon Entropy of a string of nucleotides [nt]. Since their are 4 nt, ...
0
votes
0answers
33 views

Zero conditional entropy

This question is related to this math.se question but I need a bit more guidance. For two discrete random variables $X,Y$ we define their conditional entropy to be $$H(X|Y) = -\sum_{y \in Y} Pr[Y = ...
0
votes
1answer
23 views

Is mutual information convex in the joint distribution?

Assume some joint distribution $P(X,Y) = P(Y|X)P(X)$. It is well know that, for fixed $P(Y|X)$, mutual information is a concave function of $P(X)$ and, for fixed $P(X)$, a convex function of $P(Y|X)$ ...
1
vote
2answers
37 views

Concentration property of entropy

Let $X$ be a random variable taking its values in $A = \{a_1,\ldots,a_n\}$ such that $Pr[X = a_i] = p_i$ for all $1 \leq i \leq n.$ The entropy of $X$ is defined as $$H(X) = -\sum_{i=1}^n p_i ...
2
votes
2answers
51 views

Differential Entropy

I'm a little temporarily confused about the concept of differential entropy. It says on wikipedia that the differential entropy of a Gaussian is $\log(\sigma\sqrt{2\pi e})$. However I was thinking as ...
1
vote
0answers
27 views

Is the Library of Babel random? Does it contain information?

The Library of Babel is defined as a universe in the form of a vast library containing all possible 410-page books of a certain format and character set. However, applying two means of ...
0
votes
0answers
15 views

Why $H_{V^* \cup W^*} > H_{V \cup W}$ if $H_V$ denotes entropy of language

Let $W \subseteq X^*$ be an infinite language over a finite alphabet $X$, and define ($|w|$ denotes the length of $w \in W$) $$ H_W := \limsup_{n\to \infty} \frac{\log_{|X|} | \{ w \in w \in W, |w| = ...
1
vote
1answer
22 views

Shannon information for a set of 3 equally probable elements?

How to calculate entropy as a number of binary choices for a set of three equally probable elements? The Shannon's formula gives $\log_2(3)=1.585$. But any interpretation of binary choices gives me ...
2
votes
1answer
45 views

Entropy of $f(x)=1$

Let $f(x)$ be a probability mass function $f(x) = 1$ on $x = [0,1]$, and entropy defined as $$H(p(x)) = -\int p(x) \log_2(p(x)) \, dx$$ where $p(x)$ is a pmf. Unless I've made an arithmetic error, the ...
0
votes
1answer
46 views

Approximation for a binomial coefficient sequence summation

What is a good approximation to $$\dfrac{{\binom{k}{i}}{\binom{k}{i}}(i-1)!}{\binom{k(k-1)/2}{i}}$$ $$\dfrac{{\binom{k}{i}}{\binom{k}{i}}(i-1)!}{(2^{(\log ...
0
votes
0answers
36 views

Mutual Information: Are these two equations equal?

I'm working with Multivariate Mutual Information (MMI), specifically with three variables $(X,Y,Z)$, applied to RNA sequences. The MMI equation that I use for three variables is based on entropy ...
0
votes
1answer
36 views

Partition-based entropy of a sequence

The entropy $H$ of a discrete random variable $X$ is defined by $$H(X)=E[I(X)]=\sum_xP(x)I(x)=\sum_xP(x)\log P(x)^{-1}$$ where $x$ are the possible values of $X$, $P(x)$ is the probability of $x$, ...
0
votes
0answers
19 views

How to efficiently select a subset of elements that maximizes a certain property? (entropy)

I need to select $k$ elements from a pool containing a much larger number $N$ of elements. The selection must be done in a way that a function $h(\{z_{i_1},\ldots,z_{i_k}\})$ is maximized or ...
0
votes
1answer
27 views

How to calculate entropy from a set of samples?

entropy (information content) is defined as: $$ H(X) = \sum_{i} {\mathrm{P}(x_i)\,\mathrm{I}(x_i)} = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)} $$ This allows to calculate the entropy of a ...
3
votes
1answer
68 views

“Empirical” entropy.

Information entropy is usually defined as $$\text{I}_b({\bf p}) = -\sum_{\forall i}p_i\log_b(p_i)$$ i.e. the expected value of the negative logarithm of the probabilities. This is all good when we ...
1
vote
1answer
46 views

Is conditional entropy ever taken to be a random variable?

In probability theory, the conditional expectation $E(X|Y)$ and variance $V(X|Y)$ er usually taken to be random variables, st. the value of $E(X|Y)$ depends on what value $Y$ ends up taking. I've ...
1
vote
1answer
23 views

How does a wavelet help in compressing data

I have an understanding of how we carry out image compression by using DCT along with Huffman encoding. The next subject is wavelets. I understand that wavelets are small waves and there are different ...
0
votes
0answers
51 views

Shannon Entropy Maximization with Constraints

I have got a cumulative distribution function $F_X(x)=Pr(X<=x)$. This distribution is described by 2 parameters $\alpha, \beta$. We define $F_k$ as follows: $\forall k<=n_k, ...
3
votes
1answer
51 views

Fano's Inequality Proof

For an information theory class, I am studying the proof for Fano's inequality, i.e.: $H(P_e) + P_elog(|X|) \geq H(X|\hat{X}) \geq H(X|Y)$ Where $H(X)$ is the entropy of the random variable X ...
0
votes
1answer
30 views

Mutual Information: How these two equations are equal?

I'm a biologist trying to apply the Mutual Information (MI) to some RNA secondary structure. I know that there exists two MI equation that, mathematically, are equal: $I(X,Y) = \sum_{x,y} p(x,y) ...
1
vote
1answer
31 views

Infinite closed shift-invariant set of binary sequences with zero entropy?

Does there exist an infinite closed shift-invariant $X \subset \{0,1\}^\Bbb Z$ with zero topological entropy? How to think of an example? Will periodic points of shift$|_X$ have zero entropy?
0
votes
0answers
25 views

The difference in entropy rates between a hidden process and its observation

Let $S$ be a finite state space and $o:S\to S$ an observation function. Given a distribution $p$ on $S\times S$, consider the following optimization problem: $$\max \left[ EntropyRate(\{x_t\}) - ...
0
votes
1answer
37 views

Derivation of equation for self information

I am trying to understand how the formula I(x) = -log(p(x)) for self information was derived. From what I have read, 2 constraints were imposed on the properties ...
0
votes
1answer
36 views

Clarifying Derivation of Entropy

I'm learning about probability from the book Pattern Recognition and Machine Learning by Christopher Bishop. It includes a justification for the definition of entropy that can be summarized as: let ...
2
votes
2answers
47 views

$\log (A + \delta A) = ?$ (as an expansion in $\delta A$), where $A$ and $\delta A $ are matrices

$A$ and $\delta A$ are two non-commuting matrices and I am seeking a power series expansion to 2nd order in $\delta A$. After writing it as $\log (A (1 + A^{-1}\delta A) )$, I am unable to figure out ...
2
votes
0answers
15 views

Graph Entropy - A Tractable Measure to Measure Distinguishability of Neighbourhoods

Given a labelled directed graph G, I am interested in a measurement of G that captures how distinguishable arbitrary connected sub graphs of G are. Labels may repeat and as such two or more different ...
0
votes
0answers
12 views

How to define a one-parameter family of probability distributions

I am trying to evaluate a noise-source as a means of providing entropy to a random number generator. I am running into trouble when it comes to determining the probability distribution that has the ...
1
vote
2answers
42 views

How much information does learning this interval give you?

Let's say you have a number $x$, and a priori, you know that $x \in [0, 1)$ (each value from 0 to 1 is equally likely.) Then a wizard comes and tells you that $x \in [a, b) \subseteq [0, 1)$. How much ...
1
vote
2answers
51 views

Scalar derivative of ${\rm tr}~[A(x)\log A(x)]$ where $A(x)$ is a square matrix

How do i proceed to calculate $$\frac{d}{dx}{\rm tr}\left[{A(x) \log A(x)}\right]$$ where $A(x) \in \mathbb{M}(n)$ and $x \in \mathbb{R}$? The $\log$ function is the one defined by the exponential ...
1
vote
2answers
65 views

How many bit-flips are required to achieve random distribution?

If I have a binary number W bits wide, initially all set to zero, and I repeatedly pick a random bit and toggle it from zero to one or vice versa, how many times would I need to do this to achieve ...
0
votes
1answer
63 views

mutual information adds along path

Is it true that $I(X;Y)+I(Y;Z)=I(X;Z)$ for $X \to Y \to Z$? $I(X;Z) = H(X)+H(Z)-H(X,Z)$ and $I(X;Y)+I(Y;Z) = H(X)+H(Z)-H(Z|Y)-H(X|Y)$ Hence, we would require $-H(X,Z)=-H(Z|Y)-H(X|Y)$ -- is it true? ...
0
votes
1answer
14 views

Matrix entropy measure

I have a matrix (its dimension is $n$ x $m$) where each cell can be $0$ or $1$. I would like to calculate an "entropy" measure on it that tells me how close are the ones together or how spread they ...
-1
votes
2answers
33 views

Markov chain and conditional entropy [closed]

Markov chain (DTMC) is described by transition matrix: $$\textbf{P} = \begin{pmatrix}0 & 1\\ \frac{1}{2} & \frac{1}{2} \end{pmatrix}.$$ Initial distribution $X_1 = \left(\frac{1}{4}, ...
0
votes
1answer
45 views

For $A,B,C$ independent and normal, what is $I(A+B;\ A+C)$?

Say $A,B,C$ are mutually independent and normally distributed with zero mean but possibly different variances $\sigma_1,\sigma_2,\sigma_3$. What is the mutual information between $A+B$ and $A+C$? All ...
0
votes
0answers
12 views

Which argument in KL Divergence minimization?

The KL divergence $D_{KL}(p||q) = p^T\ln(\frac{p}{q})$ is not a distance measure because first of all it is not symmetric. In applications, one usually has a prior distribution, say $y$, and wants ...
0
votes
0answers
28 views

Logarithm of Probability measure of a set

What does the parameter K immediately suggest? Suppose we have a non-uniform probability measure $Q$ on a set of sequences of length $n$, $A$, and ${Q^n}$ is the corresponding product measure. $K = ...
1
vote
1answer
43 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove (or disprove) that $$ H(X) \leq ...
2
votes
2answers
39 views

Shannon entropy property proof

X and Y are two discrete random variables having $n$ possible values : $x_{i}(1\leq i \leq n)$ and $y_{j} (1\leq j \leq n)$. The probability mass function of X is given by $$ Pr(X=x_{i}) = p_{i}, ...
1
vote
0answers
13 views

Are there any other measures of complexity for a continuous map than topological entropy?

Let $X$ be a compact topological space and $T\colon X\to X$ be continuous. In order to say something about the complexity of $(X,T)$ there is of course the notion of topological entropy of $T$, ...
1
vote
1answer
23 views

Can topological entropy be infinte?

I wonder if the topological entropy as defined by Adler or Bowen can be infinity. Can you answer that?
1
vote
0answers
14 views

Entropy of a 2D function versus 1D function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
0
votes
0answers
44 views

Entropy of $\operatorname{Beta}(\alpha, \beta, a, c)$

I know that the differential entropy of the two parameter Beta distribution $X \sim \operatorname{Beta}(\alpha, \beta)$ is $$ \begin{align} h(X) = \ln \operatorname{B} (\alpha, \beta) &- ...
0
votes
1answer
20 views

How to prove the inequality on relative entropy?

Here is the definition of Relative Entropy Now I am only interested in the simplest condition that the index set is finite and discrete, as the naive probability distribution vectors. Now if the ...
2
votes
1answer
68 views

Topological Entropy of $T$, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B\cup C,T_{|A\cup B\cup C})$ denote the toplogical entropy of $T$, restricted on $A\cup B\cup C$, where $A,B,C\subset X$ ...
0
votes
1answer
10 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
0
votes
1answer
47 views

Joint entropy calculation of discrete random variables

Suppose that i want to calculate the joint entropy $H(A,B)$ of two discrete random variables of the form: $A=\{-1,1,1,-1,-1,-1,1,1\}$ and $B=\{1,-1,1,1,-1,-1,-1,1\}$. If the goal was just the ...
0
votes
1answer
36 views

Calculating Entropy

Hi there kind people, I'm studying for an Artificial Intelligence test in a week or so, and this question is from a past paper - and it has really stumped me. Any help would be appreciated. Thank ...
0
votes
1answer
23 views

The relation between the entropy of random variables $X$ and $Y=g(X)$

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random ...
0
votes
0answers
9 views

Entropy when left- and right shift move onto each other?

I do not know how to ask my question precisely but I try. If I have a finite alphabet $$ A=\left\{0,...,n-1\right\} $$ and then consider the space $$ A^{\mathbb{Z}} $$ with the left shift. Then for ...