This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

-1
votes
2answers
48 views

Markov chain and conditional entropy [closed]

Markov chain (DTMC) is described by transition matrix: $$\textbf{P} = \begin{pmatrix}0 & 1\\ \frac{1}{2} & \frac{1}{2} \end{pmatrix}.$$ Initial distribution $X_1 = \left(\frac{1}{4}, ...
0
votes
1answer
47 views

For $A,B,C$ independent and normal, what is $I(A+B;\ A+C)$?

Say $A,B,C$ are mutually independent and normally distributed with zero mean but possibly different variances $\sigma_1,\sigma_2,\sigma_3$. What is the mutual information between $A+B$ and $A+C$? All ...
0
votes
0answers
22 views

Which argument in KL Divergence minimization?

The KL divergence $D_{KL}(p||q) = p^T\ln(\frac{p}{q})$ is not a distance measure because first of all it is not symmetric. In applications, one usually has a prior distribution, say $y$, and wants ...
1
vote
1answer
49 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove (or disprove) that $$ H(X) \leq ...
2
votes
2answers
94 views

Shannon entropy property proof

X and Y are two discrete random variables having $n$ possible values : $x_{i}(1\leq i \leq n)$ and $y_{j} (1\leq j \leq n)$. The probability mass function of X is given by $$ Pr(X=x_{i}) = p_{i}, ...
1
vote
0answers
13 views

Are there any other measures of complexity for a continuous map than topological entropy?

Let $X$ be a compact topological space and $T\colon X\to X$ be continuous. In order to say something about the complexity of $(X,T)$ there is of course the notion of topological entropy of $T$, ...
1
vote
1answer
36 views

Can topological entropy be infinte?

I wonder if the topological entropy as defined by Adler or Bowen can be infinity. Can you answer that?
1
vote
0answers
17 views

Entropy of a 2D function versus 1D function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
0
votes
0answers
55 views

Entropy of $\operatorname{Beta}(\alpha, \beta, a, c)$

I know that the differential entropy of the two parameter Beta distribution $X \sim \operatorname{Beta}(\alpha, \beta)$ is $$ \begin{align} h(X) = \ln \operatorname{B} (\alpha, \beta) &- ...
0
votes
1answer
26 views

How to prove the inequality on relative entropy?

Here is the definition of Relative Entropy Now I am only interested in the simplest condition that the index set is finite and discrete, as the naive probability distribution vectors. Now if the ...
2
votes
1answer
70 views

Topological Entropy of $T$, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B\cup C,T_{|A\cup B\cup C})$ denote the toplogical entropy of $T$, restricted on $A\cup B\cup C$, where $A,B,C\subset X$ ...
0
votes
1answer
13 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
0
votes
1answer
63 views

Joint entropy calculation of discrete random variables

Suppose that i want to calculate the joint entropy $H(A,B)$ of two discrete random variables of the form: $A=\{-1,1,1,-1,-1,-1,1,1\}$ and $B=\{1,-1,1,1,-1,-1,-1,1\}$. If the goal was just the ...
0
votes
1answer
39 views

Calculating Entropy

Hi there kind people, I'm studying for an Artificial Intelligence test in a week or so, and this question is from a past paper - and it has really stumped me. Any help would be appreciated. Thank ...
0
votes
1answer
30 views

The relation between the entropy of random variables $X$ and $Y=g(X)$

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random ...
0
votes
0answers
11 views

Entropy when left- and right shift move onto each other?

I do not know how to ask my question precisely but I try. If I have a finite alphabet $$ A=\left\{0,...,n-1\right\} $$ and then consider the space $$ A^{\mathbb{Z}} $$ with the left shift. Then for ...
0
votes
2answers
109 views

Entropy Formula $\sum_i p_i log(\frac{1}{p_i})$

In my algorithms course I have been introduced to the concept of entropy and data compression, mainly using huffman encoding. I am trying to understand the formula for entropy $$\sum_i p_i ...
0
votes
1answer
21 views

How to know the result of entropy function using uniform distribution set

In the entropy function here $H(s) = -\sum P(class=i|S)log_2{P(class=i|S)}$ I am trying to understand what is the domain of it's output for any input. I know that given a set where the frequency of ...
1
vote
0answers
28 views

How to choose assymetry for KL divergence?

I have two 2D probability distributions of eye movements of two different images. Suppose I call the first distribution of Image 1: $P$, and the second distribution of image 2: $Q$. Since ...
2
votes
1answer
71 views

Is the topological entropy of the “caterpillar waves” 0?

Please let me first describe the general background. The state of the system at time $t$ will be described by a scalar or phase $u=u^t$. Both $t$ and $u$ are discrete. $u$ take values from ...
0
votes
1answer
71 views

Interpreting Entropy

All you data scientists will probably know the entropy equation: $$H(p)=-\sum_{i=1}^{n}{{p}_{i}}\cdot\log_{2}{{p}_{i}}$$ And, using this, I was messing around with some compression, and calculated ...
2
votes
0answers
44 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
0
votes
1answer
38 views

Question about conditional entropy.

Jointly distributed random variables (a) and (b) are distributed on the n-element set. Let $\varepsilon = Prob(a \ne b)$ It is needed to prove that $H(a|b) \le 1 + \varepsilon log(n-1)$. I tried ...
0
votes
1answer
44 views

Proof of an inequality with entropy and mutual information.

Entropy of a random variable (a) is (h) : $H(a) = h$. Mutual information of (a) and (b) is (3h/4) : $I(a;b) = 3h/4$. Mutual information of (a) and (c) is (3h/4) : $I(a;c) = 3h/4$. It is needed to ...
0
votes
0answers
11 views

Invariant under conjugacy: Is there a similar statement for the other inequality?

There is the following Theorem for the topological entropy $h(T)$: If $X_1,X_2$ are compact spaces and $T_i\colon X_i\to X_i$ are continuous for $i=1,2$, and if $\Phi\colon X_1\to X_2$ is a ...
2
votes
0answers
87 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a ...
1
vote
0answers
43 views

Intuition for Entropy over Fractals

Is there intuition for "mathematical" entropy. I know that physical entropy tracks the order in a dynamical system, for thermodynamics. As entropy goes up, general randomness and disorder goes up. ...
1
vote
0answers
13 views

Integrating entropy on an arbitrary boundary

Entropy, denoted as H, is: $$ H = -\int_a^b p\ln(p) dx $$ where the range a to b is some arbitrary boundary and where p is given by the classic: $$ p(x) = ...
1
vote
2answers
91 views

How to properly integrate an entropy formula?

Entropy, denoted as $H$, is $$ H = - \int_a^b f(x)\log(f(x))\mathsf dx$$ where $f$ is given by the classic: $$ f(x) = \frac1{\sigma\sqrt{2\pi}}e^{-\frac12\left(\frac{x-\mu}\sigma\right)^2}.$$ Here ...
0
votes
0answers
20 views

Why is $h(T)=\lim_{n\to\infty}\frac{1}{n}\log \#\mathcal{B}_n$?

I am reffering to this site: http://www.scholarpedia.org/article/Topological_entropy Definitionj of topological Entropy by Adler, Kohnheim For an open cover $\mathcal{U}$ of $X$, let ...
9
votes
3answers
388 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a smooth function defined on $\textbf{R}^d$. What are the assumptions I should use to assume that $$\operatorname{div}\left(\nabla G(x) +xG(x)\right)=0 \quad (\forall x\in \textbf{R}^d)$$ ...
2
votes
1answer
117 views

Two definitions of topological entropy: Why do they coincide?

I guess you all know the definition of topological entropy by using open covers for $X$ being a compact topological space and $T\colon X\to X$ being a continuous map (for example given in Walters' "An ...
1
vote
0answers
48 views

Cross-entropy minimization - equivalent unconstrained optimization problem

I'm looking at this paper "An Alternative Method for Estimating and Simulating Maximum Entropy Densities" ...
0
votes
0answers
40 views

Is that possible to calculate Shannon Entropy with a negative value of dataset?

If it's possible what is the best way to calculate it? Let say this is my dataset; (-837.96,-823.43,-822.91,-788.44,-692.69,-657.39,-656.74,-440.56,-432.43,-203.55, ...
1
vote
0answers
38 views

Proof of a fact about mutual information and entropy

It is needed to prove that for distribution (a, b, c) such that $I(a;b|c) = I(a;c|b) = I(b;c|a) = 0$ exists a random variable $d$ such that $H(d) = I(a, b, c)$ and $a,b,c$ are independent with ...
0
votes
0answers
26 views

Continuous joint entropy with fully dependent variable

Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x=y)$, where $\delta$ is a ...
1
vote
1answer
45 views

Proof of inequality with entropy

I can not prove this inequality $2H(a,b,c) \leq H(a,b) + H(a,c) + H(b,c|a),\ H-entropy $. I tried do it by using chain rule and this inequality $H(X|Y) \leq H(X;Y)$ but without any success. Please ...
0
votes
1answer
42 views

Questions Regarding Mutual Information

I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties. I've created some MATLAB code that basically makes two ...
1
vote
2answers
66 views

How do I evaluate $\mathbb E(X\log(X))$ if $X$ has a binomial distribution, for large $n$ values?

$X\sim\mathcal {Bin}(n,p)$ I want to evaluate $\sum\limits_{x=0}^n {^n\mathrm C_x} p^x(1-p)^{n-x}x\log(x)$. Is there any way to avoid the sum because my $n$ can be very large (around $10^6$)?
0
votes
0answers
24 views

Question about history of Entropy

I have started to study Ergodic theory and entropy by some books and lecture notes more than three months but unfortunately I'm not familiar with history of Entropy (I know some thing about name of ...
0
votes
1answer
40 views

a question about entropy of run length coding

I'm doing an exercise from chapter two of $\textit {elements of information theory}$. Here is the problem and its solution, . I'm not very clear about the equation 2.36 or say why does the equation ...
0
votes
0answers
35 views

Entropy of AR(1) and AR(2) model

Does anyone know any suitable papers or knowledge themselves on the steps involved in calculating how the entropy of a AR(1) or AR(2) time series model? For example, for an AR(1) process of the form: ...
1
vote
2answers
523 views

Entropy for three random variables [duplicate]

I'm just working through some information theory and entropy, and I've come into a bit of a problem. In many texts, it's easy to find the "chain rule" for entropy in two variables, and the ...
0
votes
2answers
40 views

Required bits to communicate a partial order?

Suppose that you have a ranking (i.e. a strict complete partial order) over $n$ different objects, so that the objects can be ordered as $a>b>\cdots>n$. You want to communicate the exact ...
2
votes
1answer
126 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
0
votes
0answers
47 views

Application of Jensen's Inequality. Correct?

Help would be appreciated. Consider $x \in (0,1)$ and $f(x)=x^2$ which is convex we want to show that $\mathbb{E}\Big[f(X)\Big] \geq f\Big[\mathbb{E}(X)\Big]$. Therefore, $\mathbb{E}\Big[X^{2}\Big] ...
0
votes
3answers
69 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
0
votes
0answers
19 views

Function to increase entropy for a specific number and seed and reduce it for the rest

Hello I think I am wording the title correctly. I am looking for a function / algorithm that can increase the variability or entropy of a specific number and reducing it for the rest. The function can ...
0
votes
1answer
37 views

Relative Entropy - Help please

I'm a bit stuck evaluating the relative entropy $\int_{}^{} f(\textbf{x}) \log \left(\tfrac{f(\textbf{x})}{g(\textbf{x})} \right) \mathrm{d}\textbf{x}$ (where f and g are two densities) in the case ...
1
vote
1answer
73 views

What's the maximum entropy for discrete distribution given mean and variance

I know for continuous distribution, given mean and variance, it's Normal distribution. I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance. I ...