This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

2
votes
0answers
85 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a ...
1
vote
0answers
39 views

Intuition for Entropy over Fractals

Is there intuition for "mathematical" entropy. I know that physical entropy tracks the order in a dynamical system, for thermodynamics. As entropy goes up, general randomness and disorder goes up. ...
1
vote
0answers
9 views

Integrating entropy on an arbitrary boundary

Entropy, denoted as H, is: $$ H = -\int_a^b p\ln(p) dx $$ where the range a to b is some arbitrary boundary and where p is given by the classic: $$ p(x) = ...
1
vote
2answers
62 views

How to properly integrate an entropy formula?

Entropy, denoted as $H$, is $$ H = - \int_a^b f(x)\log(f(x))\mathsf dx$$ where $f$ is given by the classic: $$ f(x) = \frac1{\sigma\sqrt{2\pi}}e^{-\frac12\left(\frac{x-\mu}\sigma\right)^2}.$$ Here ...
0
votes
0answers
16 views

Why is $h(T)=\lim_{n\to\infty}\frac{1}{n}\log \#\mathcal{B}_n$?

I am reffering to this site: http://www.scholarpedia.org/article/Topological_entropy Definitionj of topological Entropy by Adler, Kohnheim For an open cover $\mathcal{U}$ of $X$, let ...
9
votes
3answers
378 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a smooth function defined on $\textbf{R}^d$. What are the assumptions I should use to assume that $$\operatorname{div}\left(\nabla G(x) +xG(x)\right)=0 \quad (\forall x\in \textbf{R}^d)$$ ...
2
votes
1answer
98 views

Two definitions of topological entropy: Why do they coincide?

I guess you all know the definition of topological entropy by using open covers for $X$ being a compact topological space and $T\colon X\to X$ being a continuous map (for example given in Walters' "An ...
1
vote
0answers
27 views

Cross-entropy minimization - equivalent unconstrained optimization problem

I'm looking at this paper "An Alternative Method for Estimating and Simulating Maximum Entropy Densities" ...
0
votes
0answers
27 views

Is that possible to calculate Shannon Entropy with a negative value of dataset?

If it's possible what is the best way to calculate it? Let say this is my dataset; (-837.96,-823.43,-822.91,-788.44,-692.69,-657.39,-656.74,-440.56,-432.43,-203.55, ...
1
vote
0answers
33 views

Proof of a fact about mutual information and entropy

It is needed to prove that for distribution (a, b, c) such that $I(a;b|c) = I(a;c|b) = I(b;c|a) = 0$ exists a random variable $d$ such that $H(d) = I(a, b, c)$ and $a,b,c$ are independent with ...
0
votes
0answers
15 views

Continuous joint entropy with fully dependent variable

Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x=y)$, where $\delta$ is a ...
1
vote
1answer
40 views

Proof of inequality with entropy

I can not prove this inequality $2H(a,b,c) \leq H(a,b) + H(a,c) + H(b,c|a),\ H-entropy $. I tried do it by using chain rule and this inequality $H(X|Y) \leq H(X;Y)$ but without any success. Please ...
0
votes
1answer
25 views

Questions Regarding Mutual Information

I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties. I've created some MATLAB code that basically makes two ...
1
vote
2answers
56 views

How do I evaluate $\mathbb E(X\log(X))$ if $X$ has a binomial distribution, for large $n$ values?

$X\sim\mathcal {Bin}(n,p)$ I want to evaluate $\sum\limits_{x=0}^n {^n\mathrm C_x} p^x(1-p)^{n-x}x\log(x)$. Is there any way to avoid the sum because my $n$ can be very large (around $10^6$)?
1
vote
0answers
18 views

Question about history of Entropy

I have started to study Ergodic theory and entropy by some books and lecture notes more than three months but unfortunately I'm not familiar with history of Entropy (I know some thing about name of ...
0
votes
1answer
35 views

a question about entropy of run length coding

I'm doing an exercise from chapter two of $\textit {elements of information theory}$. Here is the problem and its solution, . I'm not very clear about the equation 2.36 or say why does the equation ...
0
votes
0answers
30 views

Entropy of AR(1) and AR(2) model

Does anyone know any suitable papers or knowledge themselves on the steps involved in calculating how the entropy of a AR(1) or AR(2) time series model? For example, for an AR(1) process of the form: ...
1
vote
2answers
102 views

Entropy for three random variables [duplicate]

I'm just working through some information theory and entropy, and I've come into a bit of a problem. In many texts, it's easy to find the "chain rule" for entropy in two variables, and the ...
0
votes
2answers
38 views

Required bits to communicate a partial order?

Suppose that you have a ranking (i.e. a strict complete partial order) over $n$ different objects, so that the objects can be ordered as $a>b>\cdots>n$. You want to communicate the exact ...
1
vote
1answer
60 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
0
votes
0answers
39 views

Application of Jensen's Inequality. Correct?

Help would be appreciated. Consider $x \in (0,1)$ and $f(x)=x^2$ which is convex we want to show that $\mathbb{E}\Big[f(X)\Big] \geq f\Big[\mathbb{E}(X)\Big]$. Therefore, $\mathbb{E}\Big[X^{2}\Big] ...
0
votes
3answers
37 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
0
votes
0answers
15 views

Function to increase entropy for a specific number and seed and reduce it for the rest

Hello I think I am wording the title correctly. I am looking for a function / algorithm that can increase the variability or entropy of a specific number and reducing it for the rest. The function can ...
0
votes
1answer
35 views

Relative Entropy - Help please

I'm a bit stuck evaluating the relative entropy $\int_{}^{} f(\textbf{x}) \log \left(\tfrac{f(\textbf{x})}{g(\textbf{x})} \right) \mathrm{d}\textbf{x}$ (where f and g are two densities) in the case ...
0
votes
0answers
15 views

Prove the identity $nH(X_1,…,X_n)=…$ for any $n \geq 2$

How can I prove that the identity $$nH(X_1,...,X_n)= \sum_{1\leq i_1 < i_2<...<i_n\leq n+1} H(X_{i_1},...,X_{i_n})+\sum_{i=1}^{n} H(X_i|X_j,j\neq i)$$ stands for any $n \geq 2$ For n=2 we ...
1
vote
1answer
37 views

What's the maximum entropy for discrete distribution given mean and variance

I know for continuous distribution, given mean and variance, it's Normal distribution. I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance. I ...
1
vote
2answers
36 views

For any $n\ge2$ prove that $H(X_1,X_2,…,X_n)\ge\sum\limits_{i=n}^\mathbb{n}\ H(X_i|X_j , j \neq i)$

I am trying to figure this out and I am stuck. Any ideas? For any $n\ge2$ prove that $H(X_1,X_2,\ldots,X_n)\ge\sum\limits_{i=1}^\mathbb{n}\ H(X_i\mid X_j , \ j \neq i)$
0
votes
0answers
14 views

Why differential entropy is called “differential”

The extension of the concept of entropy to continuous random variable is sometimes called continuous entropy, which makes sense, but is also often called "differential entropy". Do you know why ?
0
votes
0answers
20 views

Does there exist an uncertainty (entropy) monotonic pmf combination rule?

Assume that I have two probability mass functions (pmf's): $p:=[p_1, p_2, p_3]$ and $q:=[q_1 ,q_2, q_3]$. Further, I assume that the uncertainty of these pmf's is quantified by the Renyi quadratic ...
1
vote
0answers
18 views

Maximum entropy distribution given second order marginals

Let $p(x,y,z)$ be a probability distribution over 3 variables (suppose them discrete, but it shouldn't matter). I know that the distribution with maximal entropy which preserves the first order ...
0
votes
1answer
45 views

Ratio between forward and reverse conditional probability

I have a probability distribution $p(Z | X)$ from which I can easily sample and compute the probability at every value for $Z$ and $X$. The inverse distribution $p(X | Z)$ however can be very complex ...
2
votes
2answers
94 views

How to prove H(X,Y) $\ge$ H(Z)?

I'm solving a problem from elements of information theory, 2nd. I got stuck by question(c) and actually, I've checked the answer, here it is: How to prove the inequality from the answer that is ...
0
votes
0answers
28 views

Maximum of a function relatred to relative entropy of Gaussian Mixture Distribution

If we have the following functions \begin{equation} g(x)=\sum_{1}^{T}\frac{\epsilon_{i}}{2\pi\sigma_{i}^2}e^{-\frac{|\mathbf{x}|^2}{2\sigma_{i}^2}} \end{equation} \begin{equation} ...
0
votes
1answer
29 views

Is there any differentiable function $f$ that approximates the “entropy” of a set of numbers $S$?

Where entropy is some measure of the degree of randomness/disorder in a given set of numbers: $S = \{a_1, a_2, ..., a_i\}$ For example, the set $S_{high} = \{4,0,2,5,8,3,7,2,5\}$ has a high degree of ...
2
votes
1answer
122 views

Coupon Collector Problem for Non-Uniform Coupons: On the number of missed Coupon

Suppose $\mathcal B=\{1,2,\ldots,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,\ldots,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$. The ...
2
votes
0answers
58 views

What is the variance of self-information (or surprisal)?

The self-information of an outcome $x_i$, or surprisal, is defined as: $$ I(x_i)=-\log P(x_i), $$ where $P$ means probability. This way, the Shannon entropy can be seen as the "average" or "expected" ...
0
votes
1answer
23 views

Entropy of $Y=bX$

If I have two random variables $Y$ and $U$ related as $Y=bU$, where $b>0$ is a constant and knowing that $\text{H}(x)$ represents the shannon entropy, such that: $$ \text{H}(x)=−\int \text{p}(x) ...
0
votes
0answers
25 views

Why is ln(p_i) not rounded down in theexpression for Shannon entropy?

Entropy supposedly " is the average amount of information contained in each message received"(Wikipedia: Entropy). However, to calculate the Shannon entropy for a finite sample, we have the sum over ...
1
vote
1answer
47 views

data processing inequality using non-deterministic functions

Generally data processing inequality says that the entropy cannot increase on applying a function f, or to be precise $H(f(X))\leq H(X)$ (also it is reversed if we know the function is k-to-1 so there ...
1
vote
1answer
45 views

Entropy of a uniform distribution

The entropy of a uniform distribution is $ ln(b-a)$. With $a=0$ and $b=1$ this reduces to zero. How come there is no uncertainty?
0
votes
0answers
16 views

Change of bases for entropy

From Cover and Thomas, Elements of Information Theory: Why isn't it: $ \log_b(p) = \frac{\log_a(p)}{\log_a(b)} $, so that $ H_a(X) $ is multiplied with $ \frac{1}{\log_b a} $?
2
votes
0answers
26 views

Finding the Legendre transform of an “entropy type” functional

I want to find the Legendre Transform of $$T(f) = \int_{\mathbb{R}^2} f \log \left(\frac{f}g{}\right) \, dx$$ on a set $H_M = \{ f: f \ge 0 \text{ and } \int_{\mathbb{R}^2} f =M\}$, where g is some ...
1
vote
0answers
60 views

Entropy calculation of Fibonacci distribution

For any positive integer $N$, consider the Fibonacci sequence $F_n$ of length $N$. Using $F_n$ we can define a Fibonacci discrete probability distribution as follows: $$p_N(n)=\frac{F_n}{\sum_{k=1}^N ...
1
vote
0answers
42 views

Estimate uncertainty of a function

I have a question about estimate the uncertainty of a function. I have a variable $x$. I assume that I can predict the variable $x$ is followed by a function $f(x)$ such as $$f(x)=\exp(-x^2)$$ Now, ...
7
votes
1answer
73 views

Structure of equientropic transformations

Given a probability vector $v=(v_1,\ldots,v_n)$ with $1\geq v_i\geq 0$ and $\sum_{i=1}^n v_i=1$ its entropy can be defined as: $$ H(v):=-\sum_{i=1}^nv_i\log v_i $$ I wonder what is known about ...
0
votes
0answers
15 views

A simpler way for an entropy inequality

I have to show that $\frac {1}{N}H(X_1,...,X_N)\le H(X_1)$. for a stationary stochastic process. I know that $H(X_1,...,X_N)=\Sigma _{i=1}^N H(X_i|X_1,...,X_{i-1})$. So far I have plugged that ...
0
votes
1answer
33 views

Mutual Information Entropy Inequality

I am trying to prove $H(x,y:z)>H(x:z)+H(y:z)$ and here is what I have. LHS: $=H(xy)-H(xy|z)=-\Sigma p(xy)lg(p(xy))+\Sigma p(xy|z)lg( \frac{p(xyz)}{p(z)})$ RHS: ...
0
votes
0answers
25 views

Does anybody know of the proof of conditional entropy in general

I was wondering if someone could provide a proof of conditional entropy in general or with two and three variables or a place where I could find it. I am having trouble with some of the algebra and ...
0
votes
0answers
43 views

How to calculate the mutual information between two outputs of Rayleigh fading channels

We have the two channels: $$X_{a,i} = H_{i}s_{i} +N_{a,i} \\ X_{b,i} = H_{i}s_{i} +N_{b,i} $$ for $1 \leq i \leq n$, where $H_i$ denotes the i.i.d. channel coefficient and is a zero-mean complex ...
0
votes
1answer
19 views

Relative Entropy decomposition reference

may I ask for some reference pointers? My bad as I got a classic case of losing my reference and thus unsure what I wrote was right or wrong. I tried looking my old references and internet and didn't ...