This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

7
votes
0answers
151 views

Approximation of Shannon entropy by trigonometric functions

Define Shannon entropy by $$I(p) = -p \log_2 p$$ Numerical experimentation shows that $\sin(\pi p)^{1-1/e}$ is a good approximation to $I(p) + I(1-p)$ on $[0,1],$ never differing by more than 3.3%. ...
6
votes
0answers
61 views

At what rate does the entropy of shuffled cards converge?

Consider a somewhat primitive method of shuffling a stack of $n$ cards: In every step, take the top card and insert it at a uniformly randomly selected one of the $n$ possible positions above, between ...
5
votes
0answers
85 views

Decrease of entropy when iterating a random discrete function

Let $m$ be a positive integer. Let $S$ be the set of non-negative integers $x$ less than $m$, with $|S|=m$. Let $X_0$ be the discrete uniform distribution over $S$, with $P(x)=\begin{cases} 1/m & \...
4
votes
0answers
33 views

Arnold's combinatorial description of entropy.

V.I. Arnold says that entropy is related to the asymptotic behaviour of polynomial coefficients. This is mentioned in his book "Dynamics, Statistics and Projective Geometry of Galois Fields". Here ...
4
votes
0answers
95 views

A combinatorial problem.

Let be $(X, \mathbb{A}, \mu)$ a measure space, a partition of $ X $ is a disjoint family $\xi=\{P_1,\ldots,P_k \}$ of measurable sets such tath $\bigcup P_i=X\pmod0).$ If $\xi=\{P_1,\ldots,P_k \...
3
votes
0answers
69 views

Differential entropy of the product of Gaussian random variables

Given two independent Gaussian random variables $X \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $Y \sim \mathcal{N}(\mu_y,\sigma_y^2)$. We look at the product distribution of these two random variables $Z=...
3
votes
0answers
148 views

Very special geometric shape (No name yet?)

I suppose this geometric shape is something very 'special'. I cannot clarify in short about being 'special', but I think this shape stands together with such special shapes like the square and the ...
3
votes
0answers
124 views

Huffman codes: does less entropy imply less weighted average codeword length?

Let $\Sigma$ be a source alphabet with a probability distribution over its symbols $P$. Then, the Shannon entropy of $\Sigma$ is $$-\sum p_j \times -\mbox{log}_2(p_j)$$ where $p_j$ is the probability ...
3
votes
0answers
111 views

Topological applications of topological entropy

I just learned topological entropy during a lecture about dynamical systems, and I wonder whether there exist purely topological applications of it.
3
votes
0answers
120 views

Can one define informational content of a mathematical expression?

At least in physicist's thinking, information, vaguely, is something that allows one to select a subset from a set. Say, a system can be in states A and B, we have done a measurement on it (...
3
votes
0answers
109 views

Reference for a transformation

Has the (Lebesgue-)ergodic transformation $T: \{0,1\}^{\mathbb{N}} \to \{0,1\}^{\mathbb{N}}$ defined by $T(x(0)x(1)x(2)\cdots) = x(1)x(3)x(5)\cdots$ been well-studied? If so, where? Does it have a ...
3
votes
0answers
96 views

Which takes more energy: Shuffling a sorted deck or sorting a shuffled one?

You have an array of length $n$ containing $n$ distinct elements. You have access to a comparator on the elements (a black-box function that takes $a$ and $b$ and returns true if $a < b$, false ...
3
votes
0answers
577 views

Can I map entropy values to a common range so that they are comparable?

I am using the standard Shannon entropy formula for calculating the entropy of a system at different states. The system has a different number of possible outcomes at each state, in other words the ...
2
votes
0answers
37 views

Mathematical Definition of Entropy and a Question about the Nature of Stat. Mechanics Approach

I have been studying for quite some time now about entropic functionals, including Boltzmann-Gibbs, Renyi, Kaniadakis and Tsallis, and I am familiar with the properties that a functional has to ...
2
votes
0answers
22 views

Showing that for a family of subsets of $[n]$ enough elements appear in high frequencies

Let $\mathcal{F} \subseteq 2^{[n]}$ a familiy of subsets. Assume that the following applies: For every $A \subseteq [n]$ , such that $|A|\leq \alpha n$ ($\alpha > 0$ is given), there's a subset $...
2
votes
0answers
37 views

How to evaluate the quality of the probability distribution output of a classifier?

In a classification problem, I have trained a neural network which outputs class probabilities for a given input. For a new input, I now want to evaluate the "quality" of the neural network's ...
2
votes
0answers
41 views

Is the topological entropy of a continuous map $T\colon X\to X$ zero if $X$ is a finite topological space?

Let $X$ be a finite topological space and $T\colon X\to X$ continuous. As the title already suggests, I am wondering if the topological entropy of $T$, denoted by $h(X,T)$, then is $0$. As far as I ...
2
votes
0answers
23 views

Graph Entropy - A Tractable Measure to Measure Distinguishability of Neighbourhoods

Given a labelled directed graph G, I am interested in a measurement of G that captures how distinguishable arbitrary connected sub graphs of G are. Labels may repeat and as such two or more different ...
2
votes
0answers
47 views

Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
2
votes
0answers
89 views

Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a bi-...
2
votes
0answers
107 views

What is the variance of self-information (or surprisal)?

The self-information of an outcome $x_i$, or surprisal, is defined as: $$ I(x_i)=-\log P(x_i), $$ where $P$ means probability. This way, the Shannon entropy can be seen as the "average" or "expected" ...
2
votes
0answers
45 views

Finding the Legendre transform of an “entropy type” functional

I want to find the Legendre Transform of $$T(f) = \int_{\mathbb{R}^2} f \log \left(\frac{f}g{}\right) \, dx$$ on a set $H_M = \{ f: f \ge 0 \text{ and } \int_{\mathbb{R}^2} f =M\}$, where g is some ...
2
votes
0answers
71 views

Limit of $\frac{1}{n}\log({n\choose np})$ without using Stirling's formula

I am trying to evaluate the following limit: $$ \forall p\in(0,1) ,\lim_{n\rightarrow \infty} \frac{1}{n}\log{n\choose \lfloor np \rfloor} =H(p),$$ where $\lfloor x\rfloor$ means the greatest integer ...
2
votes
0answers
87 views

Shannon Entropy, prove $H(Wx)=H(x)+\log|\det W|$

I'm doing an essay on ICA (independent component analysis), and I could use some help. In essence, ICA is an algorithm that minimizes the entropy of $n$ $1$-dimensional random variables, but to show ...
2
votes
0answers
103 views

What connections between computer science and ergodic theory?

I have a background in ergodic theory, but I am switching to computer science/programming. I would like to know if tools from ergodic theory could be useful, especially something around coding of ...
2
votes
0answers
41 views

intuition for entropy solutions

For a hyperbolic PDE of the form $$u_t + f(u)_x = 0$$ it turns out that the right notion of solution is entropy solution. Now, the notion of classical solutions are obviously very natural, and also ...
2
votes
0answers
73 views

Maximize the expected Brier Score of P relative to P.

Fix a finite set $\mathcal S$ and let $\mathcal P$ be the collection of probability functions over the Boolean closure of $\mathcal S$. Let $\beta : \mathcal P \times \mathcal S \to \mathbb R$, with $$...
2
votes
0answers
60 views

Joint entropy maximization with a constraint

So I have 2 random variables X and Y, where X can take on values {0,1,2,3} and Y can take on values {0,1,2,3,4}. I need to maximize H(X,Y) subject to the constraint that P(X≠Y)=0.5. This also gives P(...
2
votes
0answers
76 views

Entropy of sum of random variables

Let $x_1,x_2,\dots,x_n$ by random variables which take the values $0$ or $1$ with $P(x_i = 1) = p_i$ and $P(x_i = 0) = 1-p_i$, where $0 \leq p_i \leq 1$ for $i=1,2,\dots, n$. Let $$X= \sum_{i=1}^n ...
2
votes
0answers
29 views

probability subspaces that make entropy function equal to a constant value

Given the entropy fucntion: $$ H = -\sum_i^n p(i) \ln(p(i))\,.$$ where $p(i)$ are probabilities and $n=4$, I would like to know all the points in the probability space that make $H = k$, being $k$ a ...
2
votes
0answers
224 views

How many code words if average code length equals entropy

I've been given a proof of the following: If $q\geq2$, then there is a source $S$ with $q$ symbols, and an instantaneous $r$-ary code $C$ satisfying $L(C)=H_r(S)$ if and only if $q\equiv 1 \mod{(r-1)...
2
votes
0answers
86 views

Optimization problem in the Von Neumann Entropy

I have a constrainted optimization problem in the Von Neumann Entropy. In a CVX-like syntax the problem goes as follows: given variable $\mathtt{c(n)}$ $$\begin{align} \text{minimize} \qquad & ...
2
votes
0answers
72 views

Entropy of a measure-preserving transformation

Let $T:X \rightarrow X$ be a measure-preserving transformation of the probability space $(X,B,\mu)$. ($X$ is a topological space). If $K\subset X$ is a compact and invariant set, show that $h_{\mu}(f|...
2
votes
0answers
75 views

Computing Relative entropy?

I am doing a project for my CS class and I was wondering if the following would work. I have 50 different people who have rated the same 50 books. The rating system is as follows: negative 5 = hate ...
2
votes
0answers
182 views

rényi entropy as a derivative

Let $x=(x_i)$ be a probability measure on $\{1,\ldots,n\}$. Suppose $1<p<\infty$. The Rényi entropy of $x$ is $$ H^p(x)=\frac{1}{1-p}\log \sum_{i} x_i^p. $$ Does there exist a formula for $H^p(...
2
votes
0answers
167 views

Entropy of a Finite State Transducer

Theorem 7 in Shannon's seminal paper A Mathematical Theory of Communication states: "The output of a finite state transducer driven by a finite state statistical source is a finite state ...
1
vote
0answers
55 views

Difference of Entropy of two-dimensional Gaussians

I encountered a putative contradiction. Assume we have two 2-dim. Gaussian variables $z_1 = (x_1, y_1)$ and $z_2 = (x_2, y_2)$ with all components being independent, normal distributed variables: $x_1,...
1
vote
0answers
16 views

compute the L^2-distance of a given function to the set of Gaussian functions

I am faced with the following question: given a probability density function $f$ over $\mathbb{R}$ with $\int_{\mathbb{R}}f(x)x^2dx=\sigma^2$ given, I am trying to find the "nearest" Gaussian to $\...
1
vote
0answers
33 views

Conditional entropy $H(A,B|C)$ independent of whether $p(A,B)$ factorises or not

I arrived at a (slightly different but hopefully more precise) reformulation of my original question shown below: Is $H(A,B)-H(A,B|C)$ maximal for independent $A$ and $B$ in a similar sense as $H(A,B)...
1
vote
0answers
24 views

Calculating change in entropy, Integration help?

So I need to calculate the change in entropy for a non-ideal gas over two states. The equation im looking at using is $$T\,ds = dh - v\,dP \text{ or } T\,ds = du + P\,dv$$ where $h=u+Pv$ , $s = \...
1
vote
0answers
25 views

How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
1
vote
0answers
17 views

Channel capacity of sum of symmetric channels

I've got a channel matrix $P$ of the form $\begin{bmatrix} Q \\ R \end{bmatrix}$ where $Q,R$ are channel matrices of symmetric channels, so they now have different input alphabets but the ...
1
vote
0answers
32 views

conditional mutual information

I have a question about mutual information $$I(Z ; T/X,Y) = I(T/X,Y ; Z)$$ $T,X,Y,Z$ are random variables is this statement accurate? if it is true and I know that I(Z;T/X,Y) = H(Z/X,Y) - H(Z/T,X,...
1
vote
0answers
27 views

Entropy of a dictionary

I have an english dictionary (a file that contains a list of words) and I want to calculate: given a path tree (a word), measure $H(C_{l+1}|C_l=c_l, C_{l-1}=c_{l-1}, ...)$ for a few levels of the ...
1
vote
0answers
14 views

Entropy of first char of a word in dictionary

Suppose I have an English dictionary, that is a list of words in a file. I have to calculate the entropy of the first char. I calculated the probability of each first char ($P_a, P_b, \dots, P_z$) in ...
1
vote
0answers
52 views

mutual information and data processing inequality for $X\to Y\to Z$ where $Y=f(X)$

Let $X\to Y\to Z$ be three random variables. The data processing inequality states $I(X;Y)\geq I(X;Z)$. Further assume $Y=f(X)$ where $f:\mathcal{X}\to\mathcal{Y}$ is an arbitrary function. What ...
1
vote
0answers
38 views

Multivariate Gaussian with singular covariance matrix

The entropy of a multivariate Gaussian is given at https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Entropy as $$\frac{1}{2}\ln((2\pi e)^n |\Sigma|).$$ Here $n$ is the dimension of the ...
1
vote
0answers
46 views

Entropy and the probability to guess

Let $X$ be a discrete random variable and suppose that we choose a random value $X=x_1$. Let $A$ be an event such that $H[ X \mid A] = k$, where $$H[X \mid A] = - \sum_{x} P[X =x \mid A] \log_2( P[X ...
1
vote
0answers
34 views

How to show the inf can be achieved by some nonnegative $u\in H^1(M)$?

When I read some about Perelman's $\mathcal W$ function, I get stuck with the red line in the picture below.Seemly, I should to read the 8.2 Existence of minimizers of Evans' PDE. But I am not sure , ...
1
vote
0answers
29 views

Show entropy bound unit simplex

Let $\mathcal{S}_d$ be the $d$-dimensional unit simplex. Then for the norm $||x||_1 = \sum_i |x_i|$ and $0 < \varepsilon \leq 1$, $$N(\varepsilon, \mathcal{S}_d, ||\cdot||_1) \leq \left(\frac{5}{\...