This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

2
votes
0answers
7 views

Relationship between asymptotic full cardinality and full measure

Let $A$ be a finite set and for every integer $n \geq 1$ let $B_n \subset A^n$ such that $\dfrac{\#B_n}{\#A^n} \to 1$. Now, let $(X_n)_{n \in \mathbb{Z}}$ be a stationary process on $A$. Is there a ...
0
votes
0answers
4 views

Topological Entropy of T, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B,T_{|A\cup B})$ denote the toplogical entropy of $T$, restricted on $A\cup B$. Let $A,B\subset X$ be disjoint. Is it ...
0
votes
0answers
8 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
-1
votes
0answers
9 views

In which cases is the topological entropy equal to the grwoth rate of the periodic points?

In which cases is the topological entropy equal to the grwoth rate of the periodic points? I did find several statements, f.e. that for expanding maps this is the case. Are there other cases? Or is ...
0
votes
1answer
21 views

Joint entropy calculation of discrete random variables

Suppose that i want to calculate the joint entropy $H(A,B)$ of two discrete random variables of the form: $A=\{-1,1,1,-1,-1,-1,1,1\}$ and $B=\{1,-1,1,1,-1,-1,-1,1\}$. If the goal was just the ...
0
votes
1answer
33 views

Calculating Entropy

Hi there kind people, I'm studying for an Artificial Intelligence test in a week or so, and this question is from a past paper - and it has really stumped me. Any help would be appreciated. Thank ...
0
votes
1answer
19 views

The relation between the entropy of random variables $X$ and $Y=g(X)$

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random ...
0
votes
0answers
7 views

Entropy when left- and right shift move onto each other?

I do not know how to ask my question precisely but I try. If I have a finite alphabet $$ A=\left\{0,...,n-1\right\} $$ and then consider the space $$ A^{\mathbb{Z}} $$ with the left shift. Then for ...
0
votes
2answers
34 views

Entropy Formula $\sum_i p_i log(\frac{1}{p_i})$

In my algorithms course I have been introduced to the concept of entropy and data compression, mainly using huffman encoding. I am trying to understand the formula for entropy $$\sum_i p_i ...
0
votes
1answer
12 views

How to know the result of entropy function using uniform distribution set

In the entropy function here $H(s) = -\sum P(class=i|S)log_2{P(class=i|S)}$ I am trying to understand what is the domain of it's output for any input. I know that given a set where the frequency of ...
0
votes
0answers
11 views

How to choose assymetry for KL divergence?

I have two 2D probability distributions of eye movements of two different images. Suppose I call the first distribution of Image 1: $P$, and the second distribution of image 2: $Q$. Since ...
2
votes
1answer
67 views

Is the topological entropy of the “caterpillar waves” 0?

Please let me first describe the general background. The state of the system at time $t$ will be described by a scalar or phase $u=u^t$. Both $t$ and $u$ are discrete. $u$ take values from ...
0
votes
1answer
25 views

Interpreting Entropy

All you data scientists will probably know the entropy equation: $$H(p)=-\sum_{i=1}^{n}{{p}_{i}}\cdot\log_{2}{{p}_{i}}$$ And, using this, I was messing around with some compression, and calculated ...
0
votes
0answers
15 views

Topological Entropy and generator: Do we need that T is a homeomorphism?

There is the following statement in Walters concerning the computation of Topological Entropy in case of an expansive homeomorphism: Let $T\colon X\to X$ be an expansive homeomorphism on a compact ...
1
vote
0answers
28 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
0
votes
1answer
32 views

Question about conditional entropy.

Jointly distributed random variables (a) and (b) are distributed on the n-element set. Let $\varepsilon = Prob(a \ne b)$ It is needed to prove that $H(a|b) \le 1 + \varepsilon log(n-1)$. I tried ...
0
votes
1answer
29 views

Proof of an inequality with entropy and mutual information.

Entropy of a random variable (a) is (h) : $H(a) = h$. Mutual information of (a) and (b) is (3h/4) : $I(a;b) = 3h/4$. Mutual information of (a) and (c) is (3h/4) : $I(a;c) = 3h/4$. It is needed to ...
0
votes
0answers
6 views

Invariant under conjugacy: Is there a similar statement for the other inequality?

There is the following Theorem for the topological entropy $h(T)$: If $X_1,X_2$ are compact spaces and $T_i\colon X_i\to X_i$ are continuous for $i=1,2$, and if $\Phi\colon X_1\to X_2$ is a ...
2
votes
0answers
85 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a ...
1
vote
0answers
35 views

Intuition for Entropy over Fractals

Is there intuition for "mathematical" entropy. I know that physical entropy tracks the order in a dynamical system, for thermodynamics. As entropy goes up, general randomness and disorder goes up. ...
0
votes
0answers
17 views

Mixture of Maximum Entropy and Minimum Cross Entropy?

Assume you have a discrete prior distribution on a set of points $ P(X=\{0,3,5,6\}) = (.40,.30,.20,.10)$ $E[X]=5/2$ And you want to create a new distribution, $Y$, on $\{0,1,2,3,4,...\}$ using the ...
1
vote
0answers
4 views

Integrating entropy on an arbitrary boundary

Entropy, denoted as H, is: $$ H = -\int_a^b p\ln(p) dx $$ where the range a to b is some arbitrary boundary and where p is given by the classic: $$ p(x) = ...
1
vote
2answers
56 views

How to properly integrate an entropy formula?

Entropy, denoted as $H$, is $$ H = - \int_a^b f(x)\log(f(x))\mathsf dx$$ where $f$ is given by the classic: $$ f(x) = \frac1{\sigma\sqrt{2\pi}}e^{-\frac12\left(\frac{x-\mu}\sigma\right)^2}.$$ Here ...
0
votes
0answers
16 views

Why is $h(T)=\lim_{n\to\infty}\frac{1}{n}\log \#\mathcal{B}_n$?

I am reffering to this site: http://www.scholarpedia.org/article/Topological_entropy Definitionj of topological Entropy by Adler, Kohnheim For an open cover $\mathcal{U}$ of $X$, let ...
6
votes
1answer
166 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a function of $\mathcal{C}^2(\textbf{R}^d$;$\textbf{R}^*_+)$ such that $G \in \operatorname{L}^1(\textbf{R}^d)$. I read on the Internet that one has the following equivalence ...
1
vote
1answer
90 views

Two definitions of topological entropy: Why do they coincide?

I guess you all know the definition of topological entropy by using open covers for $X$ being a compact topological space and $T\colon X\to X$ being a continuous map (for example given in Walters' "An ...
1
vote
0answers
16 views

Cross-entropy minimization - equivalent unconstrained optimization problem

I'm looking at this paper "An Alternative Method for Estimating and Simulating Maximum Entropy Densities" ...
0
votes
0answers
16 views

Is that possible to calculate Shannon Entropy with a negative value of dataset?

If it's possible what is the best way to calculate it? Let say this is my dataset; (-837.96,-823.43,-822.91,-788.44,-692.69,-657.39,-656.74,-440.56,-432.43,-203.55, ...
1
vote
0answers
29 views

Proof of a fact about mutual information and entropy

It is needed to prove that for distribution (a, b, c) such that $I(a;b|c) = I(a;c|b) = I(b;c|a) = 0$ exists a random variable $d$ such that $H(d) = I(a, b, c)$ and $a,b,c$ are independent with ...
0
votes
0answers
11 views

Continuous joint entropy with fully dependent variable

Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x=y)$, where $\delta$ is a ...
1
vote
1answer
38 views

Proof of inequality with entropy

I can not prove this inequality $2H(a,b,c) \leq H(a,b) + H(a,c) + H(b,c|a),\ H-entropy $. I tried do it by using chain rule and this inequality $H(X|Y) \leq H(X;Y)$ but without any success. Please ...
0
votes
1answer
24 views

Questions Regarding Mutual Information

I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties. I've created some MATLAB code that basically makes two ...
1
vote
2answers
52 views

How do I evaluate $\mathbb E(X\log(X))$ if $X$ has a binomial distribution, for large $n$ values?

$X\sim\mathcal {Bin}(n,p)$ I want to evaluate $\sum\limits_{x=0}^n {^n\mathrm C_x} p^x(1-p)^{n-x}x\log(x)$. Is there any way to avoid the sum because my $n$ can be very large (around $10^6$)?
1
vote
0answers
18 views

Question about history of Entropy

I have started to study Ergodic theory and entropy by some books and lecture notes more than three months but unfortunately I'm not familiar with history of Entropy (I know some thing about name of ...
0
votes
1answer
31 views

a question about entropy of run length coding

I'm doing an exercise from chapter two of $\textit {elements of information theory}$. Here is the problem and its solution, . I'm not very clear about the equation 2.36 or say why does the equation ...
0
votes
0answers
26 views

Entropy of AR(1) and AR(2) model

Does anyone know any suitable papers or knowledge themselves on the steps involved in calculating how the entropy of a AR(1) or AR(2) time series model? For example, for an AR(1) process of the form: ...
1
vote
2answers
46 views

Entropy for three random variables [duplicate]

I'm just working through some information theory and entropy, and I've come into a bit of a problem. In many texts, it's easy to find the "chain rule" for entropy in two variables, and the ...
0
votes
2answers
38 views

Required bits to communicate a partial order?

Suppose that you have a ranking (i.e. a strict complete partial order) over $n$ different objects, so that the objects can be ordered as $a>b>\cdots>n$. You want to communicate the exact ...
0
votes
0answers
35 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
0
votes
0answers
35 views

Application of Jensen's Inequality. Correct?

Help would be appreciated. Consider $x \in (0,1)$ and $f(x)=x^2$ which is convex we want to show that $\mathbb{E}\Big[f(X)\Big] \geq f\Big[\mathbb{E}(X)\Big]$. Therefore, $\mathbb{E}\Big[X^{2}\Big] ...
0
votes
3answers
31 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
0
votes
0answers
9 views

Function to increase entropy for a specific number and seed and reduce it for the rest

Hello I think I am wording the title correctly. I am looking for a function / algorithm that can increase the variability or entropy of a specific number and reducing it for the rest. The function can ...
0
votes
1answer
31 views

Relative Entropy - Help please

I'm a bit stuck evaluating the relative entropy $\int_{}^{} f(\textbf{x}) \log \left(\tfrac{f(\textbf{x})}{g(\textbf{x})} \right) \mathrm{d}\textbf{x}$ (where f and g are two densities) in the case ...
0
votes
0answers
15 views

Prove the identity $nH(X_1,…,X_n)=…$ for any $n \geq 2$

How can I prove that the identity $$nH(X_1,...,X_n)= \sum_{1\leq i_1 < i_2<...<i_n\leq n+1} H(X_{i_1},...,X_{i_n})+\sum_{i=1}^{n} H(X_i|X_j,j\neq i)$$ stands for any $n \geq 2$ For n=2 we ...
1
vote
1answer
26 views

What's the maximum entropy for discrete distribution given mean and variance

I know for continuous distribution, given mean and variance, it's Normal distribution. I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance. I ...
1
vote
2answers
34 views

For any $n\ge2$ prove that $H(X_1,X_2,…,X_n)\ge\sum\limits_{i=n}^\mathbb{n}\ H(X_i|X_j , j \neq i)$

I am trying to figure this out and I am stuck. Any ideas? For any $n\ge2$ prove that $H(X_1,X_2,\ldots,X_n)\ge\sum\limits_{i=1}^\mathbb{n}\ H(X_i\mid X_j , \ j \neq i)$
0
votes
0answers
13 views

Why differential entropy is called “differential”

The extension of the concept of entropy to continuous random variable is sometimes called continuous entropy, which makes sense, but is also often called "differential entropy". Do you know why ?
0
votes
0answers
17 views

Does there exist an uncertainty (entropy) monotonic pmf combination rule?

Assume that I have two probability mass functions (pmf's): $p:=[p_1, p_2, p_3]$ and $q:=[q_1 ,q_2, q_3]$. Further, I assume that the uncertainty of these pmf's is quantified by the Renyi quadratic ...
1
vote
0answers
16 views

Maximum entropy distribution given second order marginals

Let $p(x,y,z)$ be a probability distribution over 3 variables (suppose them discrete, but it shouldn't matter). I know that the distribution with maximal entropy which preserves the first order ...
0
votes
1answer
42 views

Ratio between forward and reverse conditional probability

I have a probability distribution $p(Z | X)$ from which I can easily sample and compute the probability at every value for $Z$ and $X$. The inverse distribution $p(X | Z)$ however can be very complex ...