This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
0answers
5 views

PRNG for compression

I'm trying to intuitively grasp information theory. You have a string of size X that contains a lot of information, say it's a movie. You have a string of size N << X which is going to be the ...
1
vote
2answers
394 views

Upper bound on the entropy of a sum two random variables

Let $X$ be a random variable such that $|X| \leq A$ almost surely, for some $A > 0$. Let $Z$ be independent of $X$ such that $Z \sim {\cal N}(0, N)$. My question is: How large can the entropy ...
0
votes
1answer
43 views

Conditional entropy and independent conditioning variables

Let $X,Y,Z,Y',Z'$ be random variables where $Y\sim Y', Z\sim Z'$, $Y$ and $Z$ are independent, while $Y'$ and $Z'$ are, in the sense that we have $p(X,Y,Z)=p(X|Y,Z)p(Y)p(Z)$ ...
5
votes
1answer
295 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
0
votes
0answers
30 views

mutual information and data processing inequality for $X\to Y\to Z$ where $Y=f(X)$

Let $X\to Y\to Z$ be three random variables. The data processing inequality states $I(X;Y)\geq I(X;Z)$. Further assume $Y=f(X)$ where $f:\mathcal{X}\to\mathcal{Y}$ is an arbitrary function. What ...
2
votes
2answers
31 views

chain rule conditional entropy

I have to prove the chain-rule for conditional entropy. I kept getting stuck on one step, so I looked up a proof and found this: \begin{align}H(Y\mid X)&= \sum_{x\in\mathcal X, y\in\mathcal ...
4
votes
1answer
50 views

References for information theoretic statistical tools

Strange statistical concepts like spaces of probability distributions, "metrics" like Fisher information or relative entropy, and convergence with respect to these quantities are necessitated in my ...
2
votes
2answers
77 views

Reference for entropy of the binomial distribution?

The Wikipedia page Binomial distribution says that the entropy of the Binomial(n,p) is $\frac{1}{2}\log_2\left(2\pi e n p (1-p)\right) + O\left(\frac{1}{n}\right)$. What is a reference (paper or ...
2
votes
0answers
28 views

Differential entropy of the product of Gaussian random variables

Given two independent Gaussian random variables $X \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $Y \sim \mathcal{N}(\mu_y,\sigma_y^2)$. We look at the product distribution of these two random variables ...
1
vote
2answers
45 views

Geometric distribution achieves maximum entropy for given mean

Let $X$ be a random variable with geometric distribution, ie $P(X=k)=p(1-p)^k$. If I calculated it correctly, $X$ has mean $E(X)=\frac{1-p}p$ and entropy $H(X)=-\log p - \frac{1-p}p\log{(1-p)}$ ...
0
votes
0answers
9 views

Books for large deviations and maximum entropy principle?

What are some good introductory resources for learning more about the theory of large deviations and the maximum entropy principle?
4
votes
1answer
466 views

Notation of cross entropy

I have a question regarding a notation that seems to be very usual. For starters, cross entropy is defined by: \begin{align}H(X, q) &= H(X) + D(p||q) \\ & =-\sum_x p(x)\log_2 q(x)\end{align} ...
2
votes
1answer
35 views

higher moments of entropy… does the variance of $log x$ have any operational meaning?

The Shannon entropy is the average of the negative log of a list of probabilities $\{ x_1 , \dots , x_d\}$, i.e. $$H(x)= -\sum\limits_{i=1}^d x_i \log x_i$$ there are of course lots of nice ...
0
votes
0answers
23 views

Is there any definition of entropy of a stochastic process?

Entropy of finite random variables is defined in Wiki https://en.wikipedia.org/wiki/Entropy_(information_theory) Entropy rate of a stochastic process is defined in Wiki ...
0
votes
1answer
50 views

What does $\operatorname{bb}()$ mean in information theory?

I'm coming across a lot of formulas in the textbook that use "bb()" however, I don't know what bb is, it isn't mentioned anywhere. For example max entropy in a binary source: $$\mathbb{H}_{max} = ...
1
vote
1answer
24 views

Relative entropy for wiener measure/wiener measure with girsanov change of drift

I've read an article on relative entropy properties that gives a result for the relative entropy of two equivalent measures as they are found in applications of girsanovs theorem. For two measures P, ...
0
votes
0answers
19 views

Comparison between Shannon's and Blackwell's measure of informativeness

I want to compare the concept of ``precision of information'' between signals $x \in X$ and states $\omega \in \Omega$ defined by Blackwell and Shannon. Denote the conditional probability ...
0
votes
0answers
22 views

Proof that the q-Gaussian is derived by maximizing Tsallis entropy

So I have to write a report paper for a course and I would like to prove that the q-Gaussian is the distribution which arises once one maximizes the Tsallis entropy. But I face difficulties in ...
3
votes
2answers
144 views

Convergence to normal distribution

Consider the probability distribution of the simple symmetric walk. That is the random variable $X_i$ equals $c$ or $-c$ with equal probability and all $X_i$ are independent and $c\geq1$. We are ...
2
votes
1answer
150 views

What is the entropy correction term for a multivariate distribution?

I wonder if anyone would be able to help me with a confusion I have got myself into please. Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a ...
1
vote
1answer
54 views

Is this some entropy I haven't heard of?

For a discrete finite probability distribution $p(s)$ the function $$\sum\limits_s p(s)\log ^2 p(s)$$ looks like the Shannon entropy but has a square on the $\log$. Is there a name for this? Or it is ...
0
votes
0answers
17 views

Finding entropy of two sets of elements H(X), H(Y) if we have conditional probability p(Y/X).

How to find entropy of two sets H(X) and H(Y) if we are given conditional probability P(Y/X) for every part of these sets? Elements of X and Y are related.
0
votes
0answers
28 views

Multivariate Gaussian with singular covariance matrix

The entropy of a multivariate Gaussian is given at https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Entropy as $$\frac{1}{2}\ln((2\pi e)^n |\Sigma|).$$ Here $n$ is the dimension of the ...
8
votes
1answer
128 views

Entropy of matrix vector product

Consider a random $n$ by $n$ matrix $A$ whose entries are chosen from $\{0,1\}$ and a random $n$ dimensional vector $x$ whose entries are also chosen from $\{0,1\}$. Assume $n$ is large. What is ...
0
votes
0answers
27 views

Approximating the set of 2-typical sequences

I have been trying to find an upper bound for the set of 2-typical sequences; here is how far I got - I would appreciate any further help very much: Let $x^n=x_1,x_2,\ldots, x_n$ be a sequence from a ...
1
vote
0answers
39 views

Entropy and the probability to guess

Let $X$ be a discrete random variable and suppose that we choose a random value $X=x_1$. Let $A$ be an event such that $H[ X \mid A] = k$, where $$H[X \mid A] = - \sum_{x} P[X =x \mid A] \log_2( P[X ...
0
votes
1answer
43 views

Entropy of a character in a String

Taking into account the Shannon entropy, I was wondering that, if we have a String like $1122344444455$ , is this possible to find out the entropy of digit $4$ in this String? In other words, I would ...
5
votes
1answer
58 views

Maximum entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only non-negative integer entries less than or equal to $n$ ...
2
votes
1answer
20 views

Entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only positive integer entries in some finite range. I am ...
1
vote
0answers
30 views

How to show the inf can be achieved by some nonnegative $u\in H^1(M)$?

When I read some about Perelman's $\mathcal W$ function, I get stuck with the red line in the picture below.Seemly, I should to read the 8.2 Existence of minimizers of Evans' PDE. But I am not sure , ...
2
votes
0answers
77 views

Optimization problem in the Von Neumann Entropy

I have a constrainted optimization problem in the Von Neumann Entropy. In a CVX-like syntax the problem goes as follows: given variable $\mathtt{c(n)}$ $$\begin{align} \text{minimize} \qquad & ...
2
votes
1answer
75 views

How to prove the function is logarithmic with coefficience

I am given a set of properties for an unknown function $f(x)$. In particular, not constantly zero, not negative, additive and continues for any $x$ between 0 and 1. I am asked to show equivalence ...
0
votes
3answers
70 views

Is there a means of calculating the entropy of a series of bits that takes correlation into account?

A common expression for calculating the entropy of a series of bits appears to be: $$-\sum_{i}{P\left (x_i\right )log_b\left (P \left (x_i\right )\right )}$$ This seems to fail (or my intuition of ...
2
votes
1answer
42 views

Compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$

The question is to compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$. First anyone knows what this distribution is? $x$ can only take ...
0
votes
0answers
29 views

Compute the entropy of Cauchy distribution

Compute the entropy of the density function $\dfrac{b}{{\pi ({b^2} + {x^2})}}$. I think the entropy of a density function $f(x)$ is given by $H = -\int f(x) \ln f(x) ~dx$ My calculation is $$H = - ...
1
vote
1answer
51 views

Mutual information for a continuous uniform distribution

I'm trying to compute using matlab the mutual information for an $ \infty $-PAM input (the limit of a very dense PAM constellation) for a range of snr and I got stuck. I'm working with a real-valued ...
3
votes
1answer
62 views

Entropy of $f(x)=1$

Let $f(x)$ be a probability density function $f(x) = 1$ on $x = [0,1]$, and entropy defined as $$H(p(x)) = -\int p(x) \log_2(p(x)) \, dx$$ where $p(x)$ is a pdf. Unless I've made an arithmetic error, ...
0
votes
0answers
14 views

Lower bound on conditional entropy over multiple random variables

I am trying to compute the best subset of features for a given random variable $X_i$ from the set of given $n$ random variables. For that I am using conditional entropy to determine the best subset, ...
1
vote
3answers
961 views

How is the formula of Shannon Entropy derived?

From this slide, it's said that the smallest possible number of bits per symbol is as the Shannon Entropy formula defined: I've read this post, and still not quite understand how is this formula ...
1
vote
0answers
20 views

Show entropy bound unit simplex

Let $\mathcal{S}_d$ be the $d$-dimensional unit simplex. Then for the norm $||x||_1 = \sum_i |x_i|$ and $0 < \varepsilon \leq 1$, $$N(\varepsilon, \mathcal{S}_d, ||\cdot||_1) \leq ...
3
votes
1answer
105 views

Special functions related to $\sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$

While doing some caculation related to von Neumann entropy, I encountered this kind of convergent series. $$\text{Exl}(x) \equiv \sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$$ In my calculation, ...
1
vote
0answers
15 views

Efficient numerical calculation of Entropy of Poisson RV

There doesn't exist a closed-form solution of following infinite series (entropy of Poisson RV): $$ \displaystyle \sum_{x=0}^{\infty} \Big( \frac{e^\lambda \lambda^x}{x!} \Big) \cdot \text{log}_2\Big( ...
2
votes
2answers
79 views

Differential Entropy

I'm a little temporarily confused about the concept of differential entropy. It says on wikipedia that the differential entropy of a Gaussian is $\log(\sigma\sqrt{2\pi e})$. However I was thinking as ...
1
vote
1answer
34 views

Finding the limit of N approaching infinity: $N(x^\frac{1}{N}-1)\approx\ln(x)+\frac{1}{2N}\ln(x)^2+…$

I am having trouble understanding the linked exercise, final paragraph (not parts a or b) Entropy Calc Problem I understand this is a physics related exercise, however, my trouble comes in at the ...
0
votes
1answer
20 views

Calculus in maximum differential entrophy

I am calculating the maximum differential entropy by lagrange multiplier. And I found this in wiki but I cannot understand why there is 1 between $ln(g(x)), ...
4
votes
0answers
27 views

Arnold's combinatorial description of entropy.

V.I. Arnold says that entropy is related to the asymptotic behaviour of polynomial coefficients. This is mentioned in his book "Dynamics, Statistics and Projective Geometry of Galois Fields". Here ...
1
vote
0answers
52 views

Mutual information of independent fair binary random variables

Let random variables $X,Y$ independent fair random variables that take the values 0 and 1 with equal probability and $Z=X+Y$. So, $I(X;Y)=0$ and I am trying to find their conditional mutual ...
0
votes
0answers
35 views

Lower Bounding postive fractions-Mutual Information

EDIT: Let $X,Y$ be random variables over some probability space with joint distribution $P$. Then the mutual information between two random variables is defined as ...
2
votes
1answer
65 views

What are differences and relationship between shannon entropy and fisher information?

When I first got into information theory, information was measured or based on shannon entropy or in other words, most books I read before were talked about shannon entropy. Today someone told me ...
3
votes
0answers
49 views

Computation of a sum using Stirling's approximation and Watson's lemma

$$Ω=\sum_{n=0}^{N-\frac{E}{\epsilon}} \frac{Ν!}{\left(\frac{N-n-\frac{E}{\epsilon}}{2}\right)!\left(\frac{N-n+\frac{E}{\epsilon}}{2}\right)!n!}$$ I am supposed to calculate the above sum using first ...