This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

3
votes
1answer
105 views

“Empirical” entropy.

Information entropy is usually defined as $$\text{I}_b({\bf p}) = -\sum_{\forall i}p_i\log_b(p_i)$$ i.e. the expected value of the negative logarithm of the probabilities. This is all good when we ...
3
votes
2answers
99 views

Is conditional entropy ever taken to be a random variable?

In probability theory, the conditional expectation $E(X|Y)$ and variance $V(X|Y)$ er usually taken to be random variables, st. the value of $E(X|Y)$ depends on what value $Y$ ends up taking. I've ...
1
vote
1answer
31 views

How does a wavelet help in compressing data

I have an understanding of how we carry out image compression by using DCT along with Huffman encoding. The next subject is wavelets. I understand that wavelets are small waves and there are different ...
0
votes
0answers
184 views

Shannon Entropy Maximization with Constraints

I have got a cumulative distribution function $F_X(x)=Pr(X<=x)$. This distribution is described by 2 parameters $\alpha, \beta$. We define $F_k$ as follows: $\forall k<=n_k, F_k=F_X(k\times\...
3
votes
1answer
195 views

Fano's Inequality Proof

For an information theory class, I am studying the proof for Fano's inequality, i.e.: $H(P_e) + P_elog(|X|) \geq H(X|\hat{X}) \geq H(X|Y)$ Where $H(X)$ is the entropy of the random variable X $\hat{...
0
votes
1answer
35 views

Mutual Information: How these two equations are equal?

I'm a biologist trying to apply the Mutual Information (MI) to some RNA secondary structure. I know that there exists two MI equation that, mathematically, are equal: $I(X,Y) = \sum_{x,y} p(x,y) log_{...
1
vote
1answer
44 views

Infinite closed shift-invariant set of binary sequences with zero entropy?

Does there exist an infinite closed shift-invariant $X \subset \{0,1\}^\Bbb Z$ with zero topological entropy? How to think of an example? Will periodic points of shift$|_X$ have zero entropy?
0
votes
0answers
36 views

The difference in entropy rates between a hidden process and its observation

Let $S$ be a finite state space and $o:S\to S$ an observation function. Given a distribution $p$ on $S\times S$, consider the following optimization problem: $$\max \left[ EntropyRate(\{x_t\}) - ...
0
votes
1answer
51 views

Derivation of equation for self information

I am trying to understand how the formula I(x) = -log(p(x)) for self information was derived. From what I have read, 2 constraints were imposed on the properties ...
0
votes
1answer
56 views

Clarifying Derivation of Entropy

I'm learning about probability from the book Pattern Recognition and Machine Learning by Christopher Bishop. It includes a justification for the definition of entropy that can be summarized as: let $...
2
votes
2answers
56 views

$\log (A + \delta A) = ?$ (as an expansion in $\delta A$), where $A$ and $\delta A $ are matrices

$A$ and $\delta A$ are two non-commuting matrices and I am seeking a power series expansion to 2nd order in $\delta A$. After writing it as $\log (A (1 + A^{-1}\delta A) )$, I am unable to figure out ...
2
votes
0answers
22 views

Graph Entropy - A Tractable Measure to Measure Distinguishability of Neighbourhoods

Given a labelled directed graph G, I am interested in a measurement of G that captures how distinguishable arbitrary connected sub graphs of G are. Labels may repeat and as such two or more different ...
1
vote
2answers
51 views

How much information does learning this interval give you?

Let's say you have a number $x$, and a priori, you know that $x \in [0, 1)$ (each value from 0 to 1 is equally likely.) Then a wizard comes and tells you that $x \in [a, b) \subseteq [0, 1)$. How much ...
1
vote
2answers
67 views

Scalar derivative of ${\rm tr}~[A(x)\log A(x)]$ where $A(x)$ is a square matrix

How do i proceed to calculate $$\frac{d}{dx}{\rm tr}\left[{A(x) \log A(x)}\right]$$ where $A(x) \in \mathbb{M}(n)$ and $x \in \mathbb{R}$? The $\log$ function is the one defined by the exponential ...
1
vote
2answers
73 views

How many bit-flips are required to achieve random distribution?

If I have a binary number W bits wide, initially all set to zero, and I repeatedly pick a random bit and toggle it from zero to one or vice versa, how many times would I need to do this to achieve ...
0
votes
1answer
67 views

mutual information adds along path

Is it true that $I(X;Y)+I(Y;Z)=I(X;Z)$ for $X \to Y \to Z$? $I(X;Z) = H(X)+H(Z)-H(X,Z)$ and $I(X;Y)+I(Y;Z) = H(X)+H(Z)-H(Z|Y)-H(X|Y)$ Hence, we would require $-H(X,Z)=-H(Z|Y)-H(X|Y)$ -- is it true? ...
0
votes
1answer
25 views

Matrix entropy measure

I have a matrix (its dimension is $n$ x $m$) where each cell can be $0$ or $1$. I would like to calculate an "entropy" measure on it that tells me how close are the ones together or how spread they ...
-2
votes
2answers
52 views

Markov chain and conditional entropy [closed]

Markov chain (DTMC) is described by transition matrix: $$\textbf{P} = \begin{pmatrix}0 & 1\\ \frac{1}{2} & \frac{1}{2} \end{pmatrix}.$$ Initial distribution $X_1 = \left(\frac{1}{4}, \frac{3}{...
0
votes
1answer
48 views

For $A,B,C$ independent and normal, what is $I(A+B;\ A+C)$?

Say $A,B,C$ are mutually independent and normally distributed with zero mean but possibly different variances $\sigma_1,\sigma_2,\sigma_3$. What is the mutual information between $A+B$ and $A+C$? All ...
1
vote
1answer
55 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove (or disprove) that $$ H(X) \leq H(...
2
votes
2answers
133 views

Shannon entropy property proof

X and Y are two discrete random variables having $n$ possible values : $x_{i}(1\leq i \leq n)$ and $y_{j} (1\leq j \leq n)$. The probability mass function of X is given by $$ Pr(X=x_{i}) = p_{i}, 1\...
1
vote
0answers
14 views

Are there any other measures of complexity for a continuous map than topological entropy?

Let $X$ be a compact topological space and $T\colon X\to X$ be continuous. In order to say something about the complexity of $(X,T)$ there is of course the notion of topological entropy of $T$, ...
1
vote
1answer
41 views

Can topological entropy be infinte?

I wonder if the topological entropy as defined by Adler or Bowen can be infinity. Can you answer that?
1
vote
0answers
22 views

Entropy of a 2-dimensional function versus 1-dimensionl function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
0
votes
1answer
29 views

How to prove the inequality on relative entropy?

Here is the definition of Relative Entropy Now I am only interested in the simplest condition that the index set is finite and discrete, as the naive probability distribution vectors. Now if the ...
2
votes
1answer
77 views

Topological Entropy of $T$, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B\cup C,T_{|A\cup B\cup C})$ denote the toplogical entropy of $T$, restricted on $A\cup B\cup C$, where $A,B,C\subset X$ ...
0
votes
1answer
15 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
0
votes
1answer
69 views

Joint entropy calculation of discrete random variables

Suppose that i want to calculate the joint entropy $H(A,B)$ of two discrete random variables of the form: $A=\{-1,1,1,-1,-1,-1,1,1\}$ and $B=\{1,-1,1,1,-1,-1,-1,1\}$. If the goal was just the ...
0
votes
1answer
46 views

Calculating Entropy

Hi there kind people, I'm studying for an Artificial Intelligence test in a week or so, and this question is from a past paper - and it has really stumped me. Any help would be appreciated. Thank ...
0
votes
1answer
34 views

The relation between the entropy of random variables $X$ and $Y=g(X)$

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random ...
0
votes
2answers
130 views

Entropy Formula $\sum_i p_i log(\frac{1}{p_i})$

In my algorithms course I have been introduced to the concept of entropy and data compression, mainly using huffman encoding. I am trying to understand the formula for entropy $$\sum_i p_i \log\left(...
0
votes
1answer
22 views

How to know the result of entropy function using uniform distribution set

In the entropy function here $H(s) = -\sum P(class=i|S)log_2{P(class=i|S)}$ I am trying to understand what is the domain of it's output for any input. I know that given a set where the frequency of ...
1
vote
0answers
29 views

How to choose assymetry for KL divergence?

I have two 2D probability distributions of eye movements of two different images. Suppose I call the first distribution of Image 1: $P$, and the second distribution of image 2: $Q$. Since KL-...
2
votes
1answer
73 views

Is the topological entropy of the “caterpillar waves” 0?

Please let me first describe the general background. The state of the system at time $t$ will be described by a scalar or phase $u=u^t$. Both $t$ and $u$ are discrete. $u$ take values from ...
0
votes
1answer
111 views

Interpreting Entropy

All you data scientists will probably know the entropy equation: $$H(p)=-\sum_{i=1}^{n}{{p}_{i}}\cdot\log_{2}{{p}_{i}}$$ And, using this, I was messing around with some compression, and calculated ...
2
votes
0answers
47 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
0
votes
1answer
41 views

Question about conditional entropy.

Jointly distributed random variables (a) and (b) are distributed on the n-element set. Let $\varepsilon = Prob(a \ne b)$ It is needed to prove that $H(a|b) \le 1 + \varepsilon log(n-1)$. I tried ...
0
votes
1answer
50 views

Proof of an inequality with entropy and mutual information.

Entropy of a random variable (a) is (h) : $H(a) = h$. Mutual information of (a) and (b) is (3h/4) : $I(a;b) = 3h/4$. Mutual information of (a) and (c) is (3h/4) : $I(a;c) = 3h/4$. It is needed to ...
2
votes
0answers
88 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a bi-...
1
vote
0answers
45 views

Intuition for Entropy over Fractals

Is there intuition for "mathematical" entropy. I know that physical entropy tracks the order in a dynamical system, for thermodynamics. As entropy goes up, general randomness and disorder goes up. ...
1
vote
0answers
14 views

Integrating entropy on an arbitrary boundary

Entropy, denoted as H, is: $$ H = -\int_a^b p\ln(p) dx $$ where the range a to b is some arbitrary boundary and where p is given by the classic: $$ p(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\...
1
vote
2answers
157 views

How to properly integrate an entropy formula?

Entropy, denoted as $H$, is $$ H = - \int_a^b f(x)\log(f(x))\mathsf dx$$ where $f$ is given by the classic: $$ f(x) = \frac1{\sigma\sqrt{2\pi}}e^{-\frac12\left(\frac{x-\mu}\sigma\right)^2}.$$ Here ...
0
votes
0answers
20 views

Why is $h(T)=\lim_{n\to\infty}\frac{1}{n}\log \#\mathcal{B}_n$?

I am reffering to this site: http://www.scholarpedia.org/article/Topological_entropy Definitionj of topological Entropy by Adler, Kohnheim For an open cover $\mathcal{U}$ of $X$, let $N(\mathcal{U})$...
9
votes
3answers
394 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a smooth function defined on $\textbf{R}^d$. What are the assumptions I should use to assume that $$\operatorname{div}\left(\nabla G(x) +xG(x)\right)=0 \quad (\forall x\in \textbf{R}^d)$$ ...
2
votes
1answer
128 views

Two definitions of topological entropy: Why do they coincide?

I guess you all know the definition of topological entropy by using open covers for $X$ being a compact topological space and $T\colon X\to X$ being a continuous map (for example given in Walters' "An ...
1
vote
0answers
73 views

Cross-entropy minimization - equivalent unconstrained optimization problem

I'm looking at this paper "An Alternative Method for Estimating and Simulating Maximum Entropy Densities" (https://editorialexpress.com/cgi-bin/conference/download.cgi?db_name=FEMES09&paper_id=780)...
1
vote
0answers
40 views

Proof of a fact about mutual information and entropy

It is needed to prove that for distribution (a, b, c) such that $I(a;b|c) = I(a;c|b) = I(b;c|a) = 0$ exists a random variable $d$ such that $H(d) = I(a, b, c)$ and $a,b,c$ are independent with ...
1
vote
1answer
55 views

Proof of inequality with entropy

I can not prove this inequality $2H(a,b,c) \leq H(a,b) + H(a,c) + H(b,c|a),\ H-entropy $. I tried do it by using chain rule and this inequality $H(X|Y) \leq H(X;Y)$ but without any success. Please ...
0
votes
1answer
53 views

Questions Regarding Mutual Information

I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties. I've created some MATLAB code that basically makes two ...
1
vote
2answers
72 views

How do I evaluate $\mathbb E(X\log(X))$ if $X$ has a binomial distribution, for large $n$ values?

$X\sim\mathcal {Bin}(n,p)$ I want to evaluate $\sum\limits_{x=0}^n {^n\mathrm C_x} p^x(1-p)^{n-x}x\log(x)$. Is there any way to avoid the sum because my $n$ can be very large (around $10^6$)?