This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
0answers
7 views

Are there any other measures of complexity for a continuous map than topological entropy?

Let $X$ be a compact topological space and $T\colon X\to X$ be continuous. In order to say something about the complexity of $(X,T)$ there is of course the notion of topological entropy of $T$, ...
6
votes
1answer
176 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a function of $\mathcal{C}^2(\textbf{R}^d$;$\textbf{R}^*_+)$ such that $G \in \operatorname{L}^1(\textbf{R}^d)$. I read on the Internet that one has the following equivalence ...
0
votes
0answers
32 views

Entropy of $\operatorname{Beta}(\alpha, \beta, a, c)$

I know that the differential entropy of the two parameter Beta distribution $X \sim \operatorname{Beta}(\alpha, \beta)$ is $$ \begin{align} h(X) = \ln \operatorname{B} (\alpha, \beta) &- ...
0
votes
1answer
18 views

Can topological entropy be infinte?

I wonder if the topological entropy as defined by Adler or Bowen can be infinity. Can you answer that?
1
vote
0answers
8 views

Entropy of a 2D function versus 1D function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
4
votes
1answer
214 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
0
votes
1answer
24 views

Decomposing factorized entropy

I am trying to figure out how the equation for factorized entropy below is derived. The equation for entropy is $H(Q) = -\sum_x Q(x)\log Q(x)$ where $Q$ is a probability distribution. Let $Q(x) = ...
0
votes
1answer
17 views

How to prove the inequality on relative entropy?

Here is the definition of Relative Entropy Now I am only interested in the simplest condition that the index set is finite and discrete, as the naive probability distribution vectors. Now if the ...
2
votes
0answers
11 views

Relationship between asymptotic full cardinality and full measure

Let $A$ be a finite set and for every integer $n \geq 1$ let $B_n \subset A^n$ such that $\dfrac{\#B_n}{\#A^n} \to 1$. Now, let $(X_n)_{n \in \mathbb{Z}}$ be a stationary process on $A$. Is there a ...
0
votes
0answers
5 views

Topological Entropy of T, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B,T_{|A\cup B})$ denote the toplogical entropy of $T$, restricted on $A\cup B$. Let $A,B\subset X$ be disjoint. Is it ...
0
votes
0answers
8 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
-1
votes
0answers
9 views

In which cases is the topological entropy equal to the grwoth rate of the periodic points?

In which cases is the topological entropy equal to the grwoth rate of the periodic points? I did find several statements, f.e. that for expanding maps this is the case. Are there other cases? Or is ...
0
votes
1answer
26 views

Joint entropy calculation of discrete random variables

Suppose that i want to calculate the joint entropy $H(A,B)$ of two discrete random variables of the form: $A=\{-1,1,1,-1,-1,-1,1,1\}$ and $B=\{1,-1,1,1,-1,-1,-1,1\}$. If the goal was just the ...
1
vote
1answer
56 views

Is there a symmetric alternative to Kullback-Leibler divergence?

I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity ...
0
votes
1answer
33 views

Calculating Entropy

Hi there kind people, I'm studying for an Artificial Intelligence test in a week or so, and this question is from a past paper - and it has really stumped me. Any help would be appreciated. Thank ...
0
votes
1answer
19 views

The relation between the entropy of random variables $X$ and $Y=g(X)$

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random ...
0
votes
2answers
36 views

Entropy Formula $\sum_i p_i log(\frac{1}{p_i})$

In my algorithms course I have been introduced to the concept of entropy and data compression, mainly using huffman encoding. I am trying to understand the formula for entropy $$\sum_i p_i ...
0
votes
0answers
8 views

Entropy when left- and right shift move onto each other?

I do not know how to ask my question precisely but I try. If I have a finite alphabet $$ A=\left\{0,...,n-1\right\} $$ and then consider the space $$ A^{\mathbb{Z}} $$ with the left shift. Then for ...
0
votes
1answer
34 views

Relative Entropy - Help please

I'm a bit stuck evaluating the relative entropy $\int_{}^{} f(\textbf{x}) \log \left(\tfrac{f(\textbf{x})}{g(\textbf{x})} \right) \mathrm{d}\textbf{x}$ (where f and g are two densities) in the case ...
0
votes
1answer
14 views

How to know the result of entropy function using uniform distribution set

In the entropy function here $H(s) = -\sum P(class=i|S)log_2{P(class=i|S)}$ I am trying to understand what is the domain of it's output for any input. I know that given a set where the frequency of ...
0
votes
0answers
12 views

How to choose assymetry for KL divergence?

I have two 2D probability distributions of eye movements of two different images. Suppose I call the first distribution of Image 1: $P$, and the second distribution of image 2: $Q$. Since ...
3
votes
1answer
81 views

toplogical entropy of general tent map

Measure theoretic entropy of General Tent maps The linked question made me wonder how to calculate the topological entropy of a general tent map. Let $I=[0,1]$ and $\alpha \in ( 0,1)$. Define $T: I ...
2
votes
1answer
67 views

Is the topological entropy of the “caterpillar waves” 0?

Please let me first describe the general background. The state of the system at time $t$ will be described by a scalar or phase $u=u^t$. Both $t$ and $u$ are discrete. $u$ take values from ...
0
votes
1answer
26 views

Interpreting Entropy

All you data scientists will probably know the entropy equation: $$H(p)=-\sum_{i=1}^{n}{{p}_{i}}\cdot\log_{2}{{p}_{i}}$$ And, using this, I was messing around with some compression, and calculated ...
4
votes
1answer
388 views

Notation of cross entropy

I have a question regarding a notation that seems to be very usual. For starters, cross entropy is defined by: \begin{align}H(X, q) &= H(X) + D(p||q) \\ & =-\sum_x p(x)\log_2 q(x)\end{align} ...
0
votes
0answers
15 views

Topological Entropy and generator: Do we need that T is a homeomorphism?

There is the following statement in Walters concerning the computation of Topological Entropy in case of an expansive homeomorphism: Let $T\colon X\to X$ be an expansive homeomorphism on a compact ...
1
vote
0answers
28 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
0
votes
1answer
32 views

Question about conditional entropy.

Jointly distributed random variables (a) and (b) are distributed on the n-element set. Let $\varepsilon = Prob(a \ne b)$ It is needed to prove that $H(a|b) \le 1 + \varepsilon log(n-1)$. I tried ...
0
votes
1answer
30 views

Proof of an inequality with entropy and mutual information.

Entropy of a random variable (a) is (h) : $H(a) = h$. Mutual information of (a) and (b) is (3h/4) : $I(a;b) = 3h/4$. Mutual information of (a) and (c) is (3h/4) : $I(a;c) = 3h/4$. It is needed to ...
2
votes
0answers
85 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a ...
0
votes
3answers
31 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
0
votes
0answers
28 views

Entropy of AR(1) and AR(2) model

Does anyone know any suitable papers or knowledge themselves on the steps involved in calculating how the entropy of a AR(1) or AR(2) time series model? For example, for an AR(1) process of the form: ...
0
votes
0answers
7 views

Invariant under conjugacy: Is there a similar statement for the other inequality?

There is the following Theorem for the topological entropy $h(T)$: If $X_1,X_2$ are compact spaces and $T_i\colon X_i\to X_i$ are continuous for $i=1,2$, and if $\Phi\colon X_1\to X_2$ is a ...
1
vote
0answers
4 views

Integrating entropy on an arbitrary boundary

Entropy, denoted as H, is: $$ H = -\int_a^b p\ln(p) dx $$ where the range a to b is some arbitrary boundary and where p is given by the classic: $$ p(x) = ...
1
vote
0answers
35 views

Intuition for Entropy over Fractals

Is there intuition for "mathematical" entropy. I know that physical entropy tracks the order in a dynamical system, for thermodynamics. As entropy goes up, general randomness and disorder goes up. ...
1
vote
2answers
56 views

How to properly integrate an entropy formula?

Entropy, denoted as $H$, is $$ H = - \int_a^b f(x)\log(f(x))\mathsf dx$$ where $f$ is given by the classic: $$ f(x) = \frac1{\sigma\sqrt{2\pi}}e^{-\frac12\left(\frac{x-\mu}\sigma\right)^2}.$$ Here ...
0
votes
0answers
19 views

Mixture of Maximum Entropy and Minimum Cross Entropy?

Assume you have a discrete prior distribution on a set of points $ P(X=\{0,3,5,6\}) = (.40,.30,.20,.10)$ $E[X]=5/2$ And you want to create a new distribution, $Y$, on $\{0,1,2,3,4,...\}$ using the ...
1
vote
1answer
90 views

Two definitions of topological entropy: Why do they coincide?

I guess you all know the definition of topological entropy by using open covers for $X$ being a compact topological space and $T\colon X\to X$ being a continuous map (for example given in Walters' "An ...
0
votes
0answers
16 views

Why is $h(T)=\lim_{n\to\infty}\frac{1}{n}\log \#\mathcal{B}_n$?

I am reffering to this site: http://www.scholarpedia.org/article/Topological_entropy Definitionj of topological Entropy by Adler, Kohnheim For an open cover $\mathcal{U}$ of $X$, let ...
1
vote
0answers
18 views

Cross-entropy minimization - equivalent unconstrained optimization problem

I'm looking at this paper "An Alternative Method for Estimating and Simulating Maximum Entropy Densities" ...
0
votes
0answers
19 views

Is that possible to calculate Shannon Entropy with a negative value of dataset?

If it's possible what is the best way to calculate it? Let say this is my dataset; (-837.96,-823.43,-822.91,-788.44,-692.69,-657.39,-656.74,-440.56,-432.43,-203.55, ...
1
vote
0answers
29 views

Proof of a fact about mutual information and entropy

It is needed to prove that for distribution (a, b, c) such that $I(a;b|c) = I(a;c|b) = I(b;c|a) = 0$ exists a random variable $d$ such that $H(d) = I(a, b, c)$ and $a,b,c$ are independent with ...
0
votes
0answers
11 views

Continuous joint entropy with fully dependent variable

Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x=y)$, where $\delta$ is a ...
0
votes
1answer
24 views

Questions Regarding Mutual Information

I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties. I've created some MATLAB code that basically makes two ...
1
vote
1answer
38 views

Proof of inequality with entropy

I can not prove this inequality $2H(a,b,c) \leq H(a,b) + H(a,c) + H(b,c|a),\ H-entropy $. I tried do it by using chain rule and this inequality $H(X|Y) \leq H(X;Y)$ but without any success. Please ...
1
vote
2answers
52 views

How do I evaluate $\mathbb E(X\log(X))$ if $X$ has a binomial distribution, for large $n$ values?

$X\sim\mathcal {Bin}(n,p)$ I want to evaluate $\sum\limits_{x=0}^n {^n\mathrm C_x} p^x(1-p)^{n-x}x\log(x)$. Is there any way to avoid the sum because my $n$ can be very large (around $10^6$)?
1
vote
0answers
18 views

Question about history of Entropy

I have started to study Ergodic theory and entropy by some books and lecture notes more than three months but unfortunately I'm not familiar with history of Entropy (I know some thing about name of ...
0
votes
1answer
32 views

a question about entropy of run length coding

I'm doing an exercise from chapter two of $\textit {elements of information theory}$. Here is the problem and its solution, . I'm not very clear about the equation 2.36 or say why does the equation ...
1
vote
2answers
51 views

Entropy for three random variables [duplicate]

I'm just working through some information theory and entropy, and I've come into a bit of a problem. In many texts, it's easy to find the "chain rule" for entropy in two variables, and the ...
0
votes
0answers
36 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...