# Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

4k views

320 views

### Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\,$ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
100 views

### “Empirical” entropy.

Information entropy is usually defined as $$\text{I}_b({\bf p}) = -\sum_{\forall i}p_i\log_b(p_i)$$ i.e. the expected value of the negative logarithm of the probabilities. This is all good when we ...
97 views

### Differential Entropy

I'm a little temporarily confused about the concept of differential entropy. It says on wikipedia that the differential entropy of a Gaussian is $\log(\sigma\sqrt{2\pi e})$. However I was thinking as ...
896 views

### equivalence between uniform and normal distribution

The principle of insufficient reason says that all outcomes are equiprobable when we have no knowledge to guess otherwise. I understand this and that this corresponds to uniform distribution. However, ...
150 views

### Approximation of Shannon entropy by trigonometric functions

Define Shannon entropy by $$I(p) = -p \log_2 p$$ Numerical experimentation shows that $\sin(\pi p)^{1-1/e}$ is a good approximation to $I(p) + I(1-p)$ on $[0,1],$ never differing by more than 3.3%. ...
2k views

### Derivation of the maximum entropy distribution

I am reading a book and having trouble following something. The problem is to try to maximize the differential entropy $-\int_{0}^{\infty}p(r)\log p(r)$ under the constraints that ...
2k views

### Inverse of binary entropy function for $0 \le x \le \frac{1}{2}$

I'm trying to find the inverse of $H_2(x) = -x \log_2 x - (1-x) \log_2 (1-x)$[1] subject to $0 \le x \le \frac{1}{2}$. This is for a computation, so an approximation is good enough. My approach was ...
168 views

Let $(X,A,\nu)$ be a probability space and $T\colon X\to X$ a measure preserving transformation $\nu$. Take a measurable partition $P=\{P_0,\dots,P_{k-1}\}$. Let$I$ be a set of all possible ...
160 views

### when it is conditional entropy minimized?

for example let us consider following table know that entropy of variable is maximum when it is equally distributed,all of it's variable has equal probability,but what about joint entropy or ...
325 views

### How can I determine the upper limit of Shannon Entropy?

I know that the maximum possible Shannon Entropy for an alphabet $X$ is $\log|X|$, where Shannon Entropy is: $$H(X) = - \sum_{x \in X} \; p(x) \log p(x)$$ but how is this upper limit computed?
156 views

### Convergence to normal distribution

Consider the probability distribution of the simple symmetric walk. That is the random variable $X_i$ equals $c$ or $-c$ with equal probability and all $X_i$ are independent and $c\geq1$. We are ...
108 views

### Special functions related to $\sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$

While doing some caculation related to von Neumann entropy, I encountered this kind of convergent series. $$\text{Exl}(x) \equiv \sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$$ In my calculation, ...
58 views

### Do the pth powers of $p$-norms define the same partial ordering on the set of all probability distributions for all $p>1$?

Consider the $p$-th power of the Schatten $p$-norm $||q||_p$ of a probability distribution $q$ , ie, the function $\sum_j q_j^p$, where $\sum_j q_j = 1$ and $q_j \geq 0$. For fixed $q$ and $p>1$ ...
133 views

### convergence of entropy and sigma-fields

This question is related to this one. Let $(X_1, X_2, \ldots)$ be a sequence of random variables such that each $X_n$ takes its values in a finite space, say $\{0,1\}$, and the $\sigma$-field ...
48 views

### Is there a statistical measure of bitwise entropy?

(Somewhat inspired by this website, particularly Section III. Also, I might be using a different definition of entropy than usual; what I am using is closest to the physics definition (the one I ...
64 views

### Entropy Solution of $u_t+(u^2/2)_x=0$

Given the initial data $$g(x)= \cases{ 1 & x< -1 \\ 0 & -1 < x< 0 \\ 2 & 0 < x< 1 \\ 0 & 1 < x \\ }$$ What is the entropy solution of $u_t+(u^2/2)_x=0$?