# Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

151 views

### Approximation of Shannon entropy by trigonometric functions

Define Shannon entropy by $$I(p) = -p \log_2 p$$ Numerical experimentation shows that $\sin(\pi p)^{1-1/e}$ is a good approximation to $I(p) + I(1-p)$ on $[0,1],$ never differing by more than 3.3%. ...
61 views

### At what rate does the entropy of shuffled cards converge?

Consider a somewhat primitive method of shuffling a stack of $n$ cards: In every step, take the top card and insert it at a uniformly randomly selected one of the $n$ possible positions above, between ...
85 views

69 views

37 views

### How to evaluate the quality of the probability distribution output of a classifier?

In a classification problem, I have trained a neural network which outputs class probabilities for a given input. For a new input, I now want to evaluate the "quality" of the neural network's ...
41 views

### Is the topological entropy of a continuous map $T\colon X\to X$ zero if $X$ is a finite topological space?

Let $X$ be a finite topological space and $T\colon X\to X$ continuous. As the title already suggests, I am wondering if the topological entropy of $T$, denoted by $h(X,T)$, then is $0$. As far as I ...
23 views

### Graph Entropy - A Tractable Measure to Measure Distinguishability of Neighbourhoods

Given a labelled directed graph G, I am interested in a measurement of G that captures how distinguishable arbitrary connected sub graphs of G are. Labels may repeat and as such two or more different ...
47 views

### Topological Entropy of $T$ on subset $Y\subset X$

Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the following dynamics described by $T\colon X\to X$ as follows: A 1 becomes a 2, a 2 becomes a 0 and a 0 becomes a 1 if at least one of its two ...
89 views

### Is it true that $\sup_{y'\in Y'}h_d(T,U^{-1}(y'))=0$, i.e. $h(T')=h(T)$? (Bowen, Topological entropy)

First I have to give the background to my question: Let $X=\left\{0,1,2\right\}^{\mathbb{Z}}$ and on it the map $T\colon X\to X$ which describes the following dynamics: For $x\in X$, which is a bi-...
107 views

### What is the variance of self-information (or surprisal)?

The self-information of an outcome $x_i$, or surprisal, is defined as: $$I(x_i)=-\log P(x_i),$$ where $P$ means probability. This way, the Shannon entropy can be seen as the "average" or "expected" ...
45 views

### Finding the Legendre transform of an “entropy type” functional

I want to find the Legendre Transform of $$T(f) = \int_{\mathbb{R}^2} f \log \left(\frac{f}g{}\right) \, dx$$ on a set $H_M = \{ f: f \ge 0 \text{ and } \int_{\mathbb{R}^2} f =M\}$, where g is some ...
71 views

### Limit of $\frac{1}{n}\log({n\choose np})$ without using Stirling's formula

I am trying to evaluate the following limit: $$\forall p\in(0,1) ,\lim_{n\rightarrow \infty} \frac{1}{n}\log{n\choose \lfloor np \rfloor} =H(p),$$ where $\lfloor x\rfloor$ means the greatest integer ...
87 views

### Shannon Entropy, prove $H(Wx)=H(x)+\log|\det W|$

I'm doing an essay on ICA (independent component analysis), and I could use some help. In essence, ICA is an algorithm that minimizes the entropy of $n$ $1$-dimensional random variables, but to show ...
103 views

### What connections between computer science and ergodic theory?

I have a background in ergodic theory, but I am switching to computer science/programming. I would like to know if tools from ergodic theory could be useful, especially something around coding of ...
41 views

### intuition for entropy solutions

For a hyperbolic PDE of the form $$u_t + f(u)_x = 0$$ it turns out that the right notion of solution is entropy solution. Now, the notion of classical solutions are obviously very natural, and also ...
73 views

Fix a finite set $\mathcal S$ and let $\mathcal P$ be the collection of probability functions over the Boolean closure of $\mathcal S$. Let $\beta : \mathcal P \times \mathcal S \to \mathbb R$, with $$... 0answers 60 views ### Joint entropy maximization with a constraint So I have 2 random variables X and Y, where X can take on values {0,1,2,3} and Y can take on values {0,1,2,3,4}. I need to maximize H(X,Y) subject to the constraint that P(Xâ‰ Y)=0.5. This also gives P(... 0answers 76 views ### Entropy of sum of random variables Let x_1,x_2,\dots,x_n by random variables which take the values 0 or 1 with P(x_i = 1) = p_i and P(x_i = 0) = 1-p_i, where 0 \leq p_i \leq 1 for i=1,2,\dots, n. Let$$X= \sum_{i=1}^n ...
Given the entropy fucntion: $$H = -\sum_i^n p(i) \ln(p(i))\,.$$ where $p(i)$ are probabilities and $n=4$, I would like to know all the points in the probability space that make $H = k$, being $k$ a ...