This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
1answer
29 views

Conditional entropy and independent conditioning variables

Let $X,Y,Z,Y',Z'$ be random variables where $Y\sim Y', Z\sim Z'$, $Y$ and $Z$ are independent, while $Y'$ and $Z'$ are, in the sense that we have $p(X,Y,Z)=p(X|Y,Z)p(Y)p(Z)$ ...
0
votes
0answers
25 views

mutual information and data processing inequality for $X\to Y\to Z$ where $Y=f(X)$

Let $X\to Y\to Z$ be three random variables. The data processing inequality states $I(X;Y)\geq I(X;Z)$. Further assume $Y=f(X)$ where $f:\mathcal{X}\to\mathcal{Y}$ is an arbitrary function. What ...
2
votes
2answers
30 views

chain rule conditional entropy

I have to prove the chain-rule for conditional entropy. I kept getting stuck on one step, so I looked up a proof and found this: \begin{align}H(Y\mid X)&= \sum_{x\in\mathcal X, y\in\mathcal ...
4
votes
1answer
48 views

References for information theoretic statistical tools

Strange statistical concepts like spaces of probability distributions, "metrics" like Fisher information or relative entropy, and convergence with respect to these quantities are necessitated in my ...
2
votes
0answers
28 views

Differential entropy of the product of Gaussian random variables

Given two independent Gaussian random variables $X \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $Y \sim \mathcal{N}(\mu_y,\sigma_y^2)$. We look at the product distribution of these two random variables ...
1
vote
2answers
37 views

Geometric distribution achieves maximum entropy for given mean

Let $X$ be a random variable with geometric distribution, ie $P(X=k)=p(1-p)^k$. If I calculated it correctly, $X$ has mean $E(X)=\frac{1-p}p$ and entropy $H(X)=-\log p - \frac{1-p}p\log{(1-p)}$ ...
0
votes
0answers
8 views

Books for large deviations and maximum entropy principle?

What are some good introductory resources for learning more about the theory of large deviations and the maximum entropy principle?
2
votes
1answer
33 views

higher moments of entropy… does the variance of $log x$ have any operational meaning?

The Shannon entropy is the average of the negative log of a list of probabilities $\{ x_1 , \dots , x_d\}$, i.e. $$H(x)= -\sum\limits_{i=1}^d x_i \log x_i$$ there are of course lots of nice ...
0
votes
0answers
23 views

Is there any definition of entropy of a stochastic process?

Entropy of finite random variables is defined in Wiki https://en.wikipedia.org/wiki/Entropy_(information_theory) Entropy rate of a stochastic process is defined in Wiki ...
0
votes
1answer
50 views

What does $\operatorname{bb}()$ mean in information theory?

I'm coming across a lot of formulas in the textbook that use "bb()" however, I don't know what bb is, it isn't mentioned anywhere. For example max entropy in a binary source: $$\mathbb{H}_{max} = ...
1
vote
1answer
24 views

Relative entropy for wiener measure/wiener measure with girsanov change of drift

I've read an article on relative entropy properties that gives a result for the relative entropy of two equivalent measures as they are found in applications of girsanovs theorem. For two measures P, ...
0
votes
0answers
19 views

Comparison between Shannon's and Blackwell's measure of informativeness

I want to compare the concept of ``precision of information'' between signals $x \in X$ and states $\omega \in \Omega$ defined by Blackwell and Shannon. Denote the conditional probability ...
0
votes
0answers
21 views

Proof that the q-Gaussian is derived by maximizing Tsallis entropy

So I have to write a report paper for a course and I would like to prove that the q-Gaussian is the distribution which arises once one maximizes the Tsallis entropy. But I face difficulties in ...
1
vote
1answer
54 views

Is this some entropy I haven't heard of?

For a discrete finite probability distribution $p(s)$ the function $$\sum\limits_s p(s)\log ^2 p(s)$$ looks like the Shannon entropy but has a square on the $\log$. Is there a name for this? Or it is ...
2
votes
1answer
149 views

What is the entropy correction term for a multivariate distribution?

I wonder if anyone would be able to help me with a confusion I have got myself into please. Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a ...
0
votes
0answers
17 views

Finding entropy of two sets of elements H(X), H(Y) if we have conditional probability p(Y/X).

How to find entropy of two sets H(X) and H(Y) if we are given conditional probability P(Y/X) for every part of these sets? Elements of X and Y are related.
0
votes
0answers
28 views

Multivariate Gaussian with singular covariance matrix

The entropy of a multivariate Gaussian is given at https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Entropy as $$\frac{1}{2}\ln((2\pi e)^n |\Sigma|).$$ Here $n$ is the dimension of the ...
3
votes
2answers
144 views

Convergence to normal distribution

Consider the probability distribution of the simple symmetric walk. That is the random variable $X_i$ equals $c$ or $-c$ with equal probability and all $X_i$ are independent and $c\geq1$. We are ...
0
votes
0answers
25 views

Approximating the set of 2-typical sequences

I have been trying to find an upper bound for the set of 2-typical sequences; here is how far I got - I would appreciate any further help very much: Let $x^n=x_1,x_2,\ldots, x_n$ be a sequence from a ...
1
vote
0answers
39 views

Entropy and the probability to guess

Let $X$ be a discrete random variable and suppose that we choose a random value $X=x_1$. Let $A$ be an event such that $H[ X \mid A] = k$, where $$H[X \mid A] = - \sum_{x} P[X =x \mid A] \log_2( P[X ...
8
votes
1answer
126 views

Entropy of matrix vector product

Consider a random $n$ by $n$ matrix $A$ whose entries are chosen from $\{0,1\}$ and a random $n$ dimensional vector $x$ whose entries are also chosen from $\{0,1\}$. Assume $n$ is large. What is ...
0
votes
1answer
43 views

Entropy of a character in a String

Taking into account the Shannon entropy, I was wondering that, if we have a String like $1122344444455$ , is this possible to find out the entropy of digit $4$ in this String? In other words, I would ...
5
votes
1answer
58 views

Maximum entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only non-negative integer entries less than or equal to $n$ ...
2
votes
1answer
20 views

Entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only positive integer entries in some finite range. I am ...
1
vote
0answers
30 views

How to show the inf can be achieved by some nonnegative $u\in H^1(M)$?

When I read some about Perelman's $\mathcal W$ function, I get stuck with the red line in the picture below.Seemly, I should to read the 8.2 Existence of minimizers of Evans' PDE. But I am not sure , ...
2
votes
1answer
42 views

Compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$

The question is to compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$. First anyone knows what this distribution is? $x$ can only take ...
0
votes
0answers
28 views

Compute the entropy of Cauchy distribution

Compute the entropy of the density function $\dfrac{b}{{\pi ({b^2} + {x^2})}}$. I think the entropy of a density function $f(x)$ is given by $H = -\int f(x) \ln f(x) ~dx$ My calculation is $$H = - ...
1
vote
1answer
50 views

Mutual information for a continuous uniform distribution

I'm trying to compute using matlab the mutual information for an $ \infty $-PAM input (the limit of a very dense PAM constellation) for a range of snr and I got stuck. I'm working with a real-valued ...
0
votes
0answers
14 views

Lower bound on conditional entropy over multiple random variables

I am trying to compute the best subset of features for a given random variable $X_i$ from the set of given $n$ random variables. For that I am using conditional entropy to determine the best subset, ...
1
vote
0answers
18 views

Show entropy bound unit simplex

Let $\mathcal{S}_d$ be the $d$-dimensional unit simplex. Then for the norm $||x||_1 = \sum_i |x_i|$ and $0 < \varepsilon \leq 1$, $$N(\varepsilon, \mathcal{S}_d, ||\cdot||_1) \leq ...
1
vote
0answers
15 views

Efficient numerical calculation of Entropy of Poisson RV

There doesn't exist a closed-form solution of following infinite series (entropy of Poisson RV): $$ \displaystyle \sum_{x=0}^{\infty} \Big( \frac{e^\lambda \lambda^x}{x!} \Big) \cdot \text{log}_2\Big( ...
3
votes
1answer
105 views

Special functions related to $\sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$

While doing some caculation related to von Neumann entropy, I encountered this kind of convergent series. $$\text{Exl}(x) \equiv \sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$$ In my calculation, ...
1
vote
1answer
34 views

Finding the limit of N approaching infinity: $N(x^\frac{1}{N}-1)\approx\ln(x)+\frac{1}{2N}\ln(x)^2+…$

I am having trouble understanding the linked exercise, final paragraph (not parts a or b) Entropy Calc Problem I understand this is a physics related exercise, however, my trouble comes in at the ...
0
votes
1answer
20 views

Calculus in maximum differential entrophy

I am calculating the maximum differential entropy by lagrange multiplier. And I found this in wiki but I cannot understand why there is 1 between $ln(g(x)), ...
4
votes
0answers
27 views

Arnold's combinatorial description of entropy.

V.I. Arnold says that entropy is related to the asymptotic behaviour of polynomial coefficients. This is mentioned in his book "Dynamics, Statistics and Projective Geometry of Galois Fields". Here ...
0
votes
0answers
35 views

Lower Bounding postive fractions-Mutual Information

EDIT: Let $X,Y$ be random variables over some probability space with joint distribution $P$. Then the mutual information between two random variables is defined as ...
2
votes
1answer
64 views

What are differences and relationship between shannon entropy and fisher information?

When I first got into information theory, information was measured or based on shannon entropy or in other words, most books I read before were talked about shannon entropy. Today someone told me ...
3
votes
0answers
49 views

Computation of a sum using Stirling's approximation and Watson's lemma

$$Ω=\sum_{n=0}^{N-\frac{E}{\epsilon}} \frac{Ν!}{\left(\frac{N-n-\frac{E}{\epsilon}}{2}\right)!\left(\frac{N-n+\frac{E}{\epsilon}}{2}\right)!n!}$$ I am supposed to calculate the above sum using first ...
1
vote
1answer
30 views

Specific conditional entropy $H(X|Y=y)$ is not bounded by $H(X)$?

Suppose that $P(Y=y)>0$ so that $$ H(X|Y=y)=-\sum_{x} p(x|y) \log_{2} p(x|y) $$ makes sense. I've assumed for a long time that $H(X|Y=y)\le H(X)$, but then it seems that the wiki article claims ...
1
vote
0answers
52 views

Mutual information of independent fair binary random variables

Let random variables $X,Y$ independent fair random variables that take the values 0 and 1 with equal probability and $Z=X+Y$. So, $I(X;Y)=0$ and I am trying to find their conditional mutual ...
0
votes
1answer
16 views

Binary symmetric channel

Binary symmetric matrix A sends $i,j$ and B gets $i,j$. Does it mean that $A$ != $B$? I would know how to solve this if A would be equal to B, but now I'm not sure how should I start, when A has 2 ...
1
vote
1answer
51 views

Finding entropy with two unknown probabilities, and maximizing it

$$S = \{ X, Y, Z, W\};\\P(X) = 0.1;\\P(Y) = 0.5;\\P(Z) = p;\\P(W) = q$$ I don't know how to find source (S) entropy with these 2 unknown $p$ and $q$ probabilities. With which $p$ and $ q$ values ...
0
votes
0answers
24 views

I choose $n$ words from $k$ randoms words from a dictionary with $t$ words. How much entropy is this password?

Let's say I have a dictionary of $t$ words. I randomly select a set of $k<t$ words (no duplicates). Next, I deterministically choose $n<k$ words from those $k$ words (say, pick the first $n$ ...
0
votes
0answers
22 views

Correlation and entropy between stocks of the same index

My portofolio contains the stocks belonging to Nasdaq100 index. Initially, i found the entropy of closing prices between Friday's value and Monday's value,for all companies of the index, in order to ...
1
vote
0answers
35 views

Asymptotic binomial ratios

I am in need of asymptotic version of $$\frac{ \displaystyle \binom{n^{1-s}}{n^s}}{\displaystyle \binom{n}{n^{s}}}$$ where $n\in\Bbb N$ and $s\in\big(0,\frac12\big)$ and $$\displaystyle \frac{ ...
1
vote
1answer
53 views

compute H(X|Y) ( conditional probablity)

Can someone help me on this? X = {$X_1,X_2,X_3,X_4$} Y = {$Y_1,Y_2,Y_3,Y_4$} Suppose p($X_i$) = p($Y_j$) = 1/4 (each X and each Y equally likely) $1 \leq i, 4 \ge j$ and now suppose $Y_1 : X_1 ...
0
votes
1answer
10 views

definition of conditional probability $(p_i|\pi_k)$ and Tsallis entropy

Let $\Omega$ be a set of $W$ possible outcomes of an experiment with probability assignments $p_i$ and thus $\sum_{i=1}^{W}p_i=1$. Now, let's divide $\Omega$ into $K$ non-intersecting subsets each ...
0
votes
1answer
30 views

Proof of chain rule for entropy of random variables

I have the following proof for the chain rule for entropy of random variables: We write: \begin{eqnarray*} ...
1
vote
0answers
28 views

Mutual information between 2 sequences of random variables?

How would I go about expanding $I(X_1,...,X_n;Y_1,...,Y_n)$? The chain rule exists for a single case, i.e.: $I(X_1,...,X_n;Y)=\sum^n_{i=1} I(X_i;Y|X_{i-1},...,X_1)$, but I'm having doubts if this can ...
2
votes
0answers
30 views

How to evaluate the quality of the probability distribution output of a classifier?

In a classification problem, I have trained a neural network which outputs class probabilities for a given input. For a new input, I now want to evaluate the "quality" of the neural network's ...