This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
0answers
16 views

Derivation of equation for self information

I am trying to understand how the formula I(x) = -log(p(x)) for self information was derived. From what I have read, 2 constraints were imposed on the properties ...
0
votes
0answers
23 views

Clarifying Derivation of Entropy

I'm learning about probability from the book Pattern Recognition and Machine Learning by Christopher Bishop. It includes a justification for the definition of entropy that can be summarized as: let ...
9
votes
3answers
367 views

Does $\operatorname{div}\left(\nabla G +xG\right)=0\Longleftrightarrow \nabla G +xG=0$?

Let $G$ be a smooth function defined on $\textbf{R}^d$. What are the assumptions I should use to assume that $$\operatorname{div}\left(\nabla G(x) +xG(x)\right)=0 \quad (\forall x\in \textbf{R}^d)$$ ...
2
votes
1answer
35 views

$\log (A + \delta A) = ?$ (as an expansion in $\delta A$), where $A$ and $\delta A $ are matrices

$A$ and $\delta A$ are two non-commuting matrices and I am seeking a power series expansion to 2nd order in $\delta A$. After writing it as $\log (A (1 + A^{-1}\delta A) )$, I am unable to figure out ...
2
votes
0answers
14 views

Graph Entropy - A Tractable Measure to Measure Distinguishability of Neighbourhoods

Given a labelled directed graph G, I am interested in a measurement of G that captures how distinguishable arbitrary connected sub graphs of G are. Labels may repeat and as such two or more different ...
0
votes
0answers
8 views

How to define a one-parameter family of probability distributions

I am trying to evaluate a noise-source as a means of providing entropy to a random number generator. I am running into trouble when it comes to determining the probability distribution that has the ...
1
vote
1answer
53 views

What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
1
vote
2answers
39 views

How much information does learning this interval give you?

Let's say you have a number $x$, and a priori, you know that $x \in [0, 1)$ (each value from 0 to 1 is equally likely.) Then a wizard comes and tells you that $x \in [a, b) \subseteq [0, 1)$. How much ...
0
votes
1answer
62 views

mutual information adds along path

Is it true that $I(X;Y)+I(Y;Z)=I(X;Z)$ for $X \to Y \to Z$? $I(X;Z) = H(X)+H(Z)-H(X,Z)$ and $I(X;Y)+I(Y;Z) = H(X)+H(Z)-H(Z|Y)-H(X|Y)$ Hence, we would require $-H(X,Z)=-H(Z|Y)-H(X|Y)$ -- is it true? ...
-1
votes
1answer
37 views

Scalar derivative of ${\rm tr}~[A(x)\log A(x)]$ where $A(x)$ is a square matrix

How do i proceed to calculate $$\frac{d}{dx}{\rm tr}\left[{A(x) \log A(x)}\right]$$ where $A(x) \in \mathbb{M}(n)$ and $x \in \mathbb{R}$? The $\log$ function is the one defined by the exponential ...
1
vote
2answers
61 views

How many bit-flips are required to achieve random distribution?

If I have a binary number W bits wide, initially all set to zero, and I repeatedly pick a random bit and toggle it from zero to one or vice versa, how many times would I need to do this to achieve ...
0
votes
1answer
14 views

Matrix entropy measure

I have a matrix (its dimension is $n$ x $m$) where each cell can be $0$ or $1$. I would like to calculate an "entropy" measure on it that tells me how close are the ones together or how spread they ...
1
vote
2answers
49 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & ...
-1
votes
2answers
28 views

Markov chain and conditional entropy [closed]

Markov chain (DTMC) is described by transition matrix: $$\textbf{P} = \begin{pmatrix}0 & 1\\ \frac{1}{2} & \frac{1}{2} \end{pmatrix}.$$ Initial distribution $X_1 = \left(\frac{1}{4}, ...
0
votes
1answer
36 views

For $A,B,C$ independent and normal, what is $I(A+B;\ A+C)$?

Say $A,B,C$ are mutually independent and normally distributed with zero mean but possibly different variances $\sigma_1,\sigma_2,\sigma_3$. What is the mutual information between $A+B$ and $A+C$? All ...
4
votes
1answer
395 views

Notation of cross entropy

I have a question regarding a notation that seems to be very usual. For starters, cross entropy is defined by: \begin{align}H(X, q) &= H(X) + D(p||q) \\ & =-\sum_x p(x)\log_2 q(x)\end{align} ...
2
votes
1answer
67 views

Topological Entropy of $T$, on a disjoint union?

Let $X$ be a compact metric space and $T\colon X\to X$ continuous. By $h(A\cup B\cup C,T_{|A\cup B\cup C})$ denote the toplogical entropy of $T$, restricted on $A\cup B\cup C$, where $A,B,C\subset X$ ...
1
vote
1answer
39 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove (or disprove) that $$ H(X) \leq ...
0
votes
0answers
6 views

Which argument in KL Divergence minimization?

The KL divergence $D_{KL}(p||q) = p^T\ln(\frac{p}{q})$ is not a distance measure because first of all it is not symmetric. In applications, one usually has a prior distribution, say $y$, and wants ...
0
votes
0answers
25 views

Logarithm of Probability measure of a set

What does the parameter K immediately suggest? Suppose we have a non-uniform probability measure $Q$ on a set of sequences of length $n$, $A$, and ${Q^n}$ is the corresponding product measure. $K = ...
1
vote
1answer
289 views

Is maximizing the Shannon differential entropy equivalent to minimizing the predictability and/or minimizing the maximum density?

For a real-valued, 1-dimensional, continuous random variable X with density f(x), I am trying to determine if maximizing the Shannon differential entropy of f(x) is mathematically equivalent to ...
2
votes
2answers
34 views

Shannon entropy property proof

X and Y are two discrete random variables having $n$ possible values : $x_{i}(1\leq i \leq n)$ and $y_{j} (1\leq j \leq n)$. The probability mass function of X is given by $$ Pr(X=x_{i}) = p_{i}, ...
1
vote
0answers
13 views

Are there any other measures of complexity for a continuous map than topological entropy?

Let $X$ be a compact topological space and $T\colon X\to X$ be continuous. In order to say something about the complexity of $(X,T)$ there is of course the notion of topological entropy of $T$, ...
0
votes
0answers
41 views

Entropy of $\operatorname{Beta}(\alpha, \beta, a, c)$

I know that the differential entropy of the two parameter Beta distribution $X \sim \operatorname{Beta}(\alpha, \beta)$ is $$ \begin{align} h(X) = \ln \operatorname{B} (\alpha, \beta) &- ...
1
vote
1answer
20 views

Can topological entropy be infinte?

I wonder if the topological entropy as defined by Adler or Bowen can be infinity. Can you answer that?
1
vote
0answers
12 views

Entropy of a 2D function versus 1D function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
4
votes
1answer
219 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
0
votes
1answer
24 views

Decomposing factorized entropy

I am trying to figure out how the equation for factorized entropy below is derived. The equation for entropy is $H(Q) = -\sum_x Q(x)\log Q(x)$ where $Q$ is a probability distribution. Let $Q(x) = ...
0
votes
1answer
18 views

How to prove the inequality on relative entropy?

Here is the definition of Relative Entropy Now I am only interested in the simplest condition that the index set is finite and discrete, as the naive probability distribution vectors. Now if the ...
0
votes
0answers
9 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
0
votes
1answer
36 views

Joint entropy calculation of discrete random variables

Suppose that i want to calculate the joint entropy $H(A,B)$ of two discrete random variables of the form: $A=\{-1,1,1,-1,-1,-1,1,1\}$ and $B=\{1,-1,1,1,-1,-1,-1,1\}$. If the goal was just the ...
1
vote
1answer
69 views

Is there a symmetric alternative to Kullback-Leibler divergence?

I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity ...
0
votes
1answer
35 views

Calculating Entropy

Hi there kind people, I'm studying for an Artificial Intelligence test in a week or so, and this question is from a past paper - and it has really stumped me. Any help would be appreciated. Thank ...
0
votes
1answer
22 views

The relation between the entropy of random variables $X$ and $Y=g(X)$

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random ...
0
votes
2answers
37 views

Entropy Formula $\sum_i p_i log(\frac{1}{p_i})$

In my algorithms course I have been introduced to the concept of entropy and data compression, mainly using huffman encoding. I am trying to understand the formula for entropy $$\sum_i p_i ...
0
votes
0answers
8 views

Entropy when left- and right shift move onto each other?

I do not know how to ask my question precisely but I try. If I have a finite alphabet $$ A=\left\{0,...,n-1\right\} $$ and then consider the space $$ A^{\mathbb{Z}} $$ with the left shift. Then for ...
0
votes
1answer
34 views

Relative Entropy - Help please

I'm a bit stuck evaluating the relative entropy $\int_{}^{} f(\textbf{x}) \log \left(\tfrac{f(\textbf{x})}{g(\textbf{x})} \right) \mathrm{d}\textbf{x}$ (where f and g are two densities) in the case ...
0
votes
1answer
15 views

How to know the result of entropy function using uniform distribution set

In the entropy function here $H(s) = -\sum P(class=i|S)log_2{P(class=i|S)}$ I am trying to understand what is the domain of it's output for any input. I know that given a set where the frequency of ...
0
votes
0answers
16 views

How to choose assymetry for KL divergence?

I have two 2D probability distributions of eye movements of two different images. Suppose I call the first distribution of Image 1: $P$, and the second distribution of image 2: $Q$. Since ...
3
votes
1answer
85 views

toplogical entropy of general tent map

Measure theoretic entropy of General Tent maps The linked question made me wonder how to calculate the topological entropy of a general tent map. Let $I=[0,1]$ and $\alpha \in ( 0,1)$. Define $T: I ...
2
votes
1answer
68 views

Is the topological entropy of the “caterpillar waves” 0?

Please let me first describe the general background. The state of the system at time $t$ will be described by a scalar or phase $u=u^t$. Both $t$ and $u$ are discrete. $u$ take values from ...
0
votes
1answer
31 views

Interpreting Entropy

All you data scientists will probably know the entropy equation: $$H(p)=-\sum_{i=1}^{n}{{p}_{i}}\cdot\log_{2}{{p}_{i}}$$ And, using this, I was messing around with some compression, and calculated ...
0
votes
0answers
16 views

Topological Entropy and generator: Do we need that T is a homeomorphism?

There is the following statement in Walters concerning the computation of Topological Entropy in case of an expansive homeomorphism: Let $T\colon X\to X$ be an expansive homeomorphism on a compact ...
2
votes
0answers
31 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
0
votes
1answer
32 views

Question about conditional entropy.

Jointly distributed random variables (a) and (b) are distributed on the n-element set. Let $\varepsilon = Prob(a \ne b)$ It is needed to prove that $H(a|b) \le 1 + \varepsilon log(n-1)$. I tried ...
0
votes
1answer
35 views

Proof of an inequality with entropy and mutual information.

Entropy of a random variable (a) is (h) : $H(a) = h$. Mutual information of (a) and (b) is (3h/4) : $I(a;b) = 3h/4$. Mutual information of (a) and (c) is (3h/4) : $I(a;c) = 3h/4$. It is needed to ...
2
votes
0answers
85 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a ...
0
votes
3answers
32 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
0
votes
0answers
29 views

Entropy of AR(1) and AR(2) model

Does anyone know any suitable papers or knowledge themselves on the steps involved in calculating how the entropy of a AR(1) or AR(2) time series model? For example, for an AR(1) process of the form: ...
0
votes
0answers
8 views

Invariant under conjugacy: Is there a similar statement for the other inequality?

There is the following Theorem for the topological entropy $h(T)$: If $X_1,X_2$ are compact spaces and $T_i\colon X_i\to X_i$ are continuous for $i=1,2$, and if $\Phi\colon X_1\to X_2$ is a ...