This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
1answer
35 views

If $X$ to $Y$ to $Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$

If $X \to Y \to Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$. I said the statement was true, and from $I(X;Y)\ge I(X;Z)$ by definition, thus $H(X) - H(X\mid Y) \ge H(X)-H(X\mid ...
1
vote
1answer
41 views

Rate distortion function with infinite distortion

I am working through the problems in Elements of Information Theory by Cover and Thomas and have come across the following problem I couldn't answer. The problem is to find the rate distortion ...
1
vote
0answers
27 views

Entropy of a dictionary

I have an english dictionary (a file that contains a list of words) and I want to calculate: given a path tree (a word), measure $H(C_{l+1}|C_l=c_l, C_{l-1}=c_{l-1}, ...)$ for a few levels of the ...
0
votes
0answers
22 views

Different methodology for maximizing entropy in continuous random variable case

Suppose we want to maximize the well-known Shannon entropy $S=-∫_{0}^{x_{max}}f(x)lnf(x)dx$ subject to the following constraints $∫_{0}^{x_{max}}f(x)dx=1$, $∫_{0}^{x_{max}}xf(x)dx=x ̅$ and so on (...
-1
votes
1answer
20 views

Does the Information Gain algorithm favor a high-entropy attribute or a low-entropy one?

This might not be mutual to mathematics but it does relate to Information-Theory. My question is: Does the InformationGain algorithm, in Decision-Tree machine-learning, favor a high-entropy ...
0
votes
0answers
32 views

Channels with memory have higher capacity

I am working through Elements of Information Theory by Cover and Thomas and have come across the following solution to one of their problems that I don't understand. Consider a binary, symmetric ...
1
vote
0answers
14 views

Entropy of first char of a word in dictionary

Suppose I have an English dictionary, that is a list of words in a file. I have to calculate the entropy of the first char. I calculated the probability of each first char ($P_a, P_b, \dots, P_z$) in ...
5
votes
1answer
184 views

Is there a possibility to determine/ estimate the topological entropy?

By $E$, denote the set of excited states $E=\left\{1,2,\ldots,e\right\}$ and by $R$ the set of refractory states $R=\left\{e+1,e+2,\ldots,e+r\right\}$. By $0$, denote the equilibrium state. The ...
1
vote
1answer
25 views

Entropy Inequality $H(X| g(Y))\geq H(X|Y)$

Let $X,Y$ be random discrete variables. $H(X) = -\sum\limits_{x}P\{X=x\}\operatorname{log}_2P\{X=x\}$ be the entropy-function. It is known fact that that $H(g(Y))\leq H(Y)$. I want to prove the ...
6
votes
0answers
59 views

At what rate does the entropy of shuffled cards converge?

Consider a somewhat primitive method of shuffling a stack of $n$ cards: In every step, take the top card and insert it at a uniformly randomly selected one of the $n$ possible positions above, between ...
0
votes
1answer
34 views

Existence of measure(s) of maximal entropy, given a finite-to-one chaotic global attractor $A$ which is, moreover, the non-wandering set

Let $X$ be a compact metric space and let $f\colon X\to X$ define some dynamics, $f$ being continuous and finite-to-one on a global attractor $A\subset X$. Moreover, $A$ is the non-wandering set $\...
0
votes
0answers
44 views

Is my entropy calculation correct? Clustering entropy example

I would like to calculate entropy of this example scheme http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html Equation of entropy Then the entropy is (the first line) ...
5
votes
1answer
97 views

Mutual information vs Information Gain

I always thought that mutual information and information gain refer to the same thing, however looking at Wikipedia: http://en.wikipedia.org/wiki/Information_gain https://en.wikipedia.org/wiki/...
0
votes
0answers
13 views

Is there an obvious explanation for why the entropy formula is better than standard deviation as a numerical summary for spread of a distribution?

I'm currently reading Murphy's Machine Learning book, and he briefly introduces the notion of entropy for a random variable. He states that the entropy is $$ \mathbb{H}(X)=-\int p(x)\log_{2}p(x)\,dx, ...
0
votes
1answer
32 views

Entropy of gamma-exponential compound distribution

Following this question, I have the PDF of a gamma-exponential compound distribution as $$f(y) = \frac{\alpha\beta^{\alpha}} {(y+\beta)^{(\alpha+1)}} $$ For my application I need the entropy of ...
0
votes
0answers
59 views

Generalization of Shannon's source coding theorem with a posteriori entropies

This doubt is with reference to section 5-5 of "Information theory and Coding" by Prof. Norman Abramson. Under the topic "A generalization of Shannon's First Theorem", the text discusses how knowledge ...
2
votes
0answers
21 views

Showing that for a family of subsets of $[n]$ enough elements appear in high frequencies

Let $\mathcal{F} \subseteq 2^{[n]}$ a familiy of subsets. Assume that the following applies: For every $A \subseteq [n]$ , such that $|A|\leq \alpha n$ ($\alpha > 0$ is given), there's a subset $...
1
vote
1answer
51 views

Calculating the capacity of a Binary Channel?

I'm pretty new in information theory so I need your suggestions or references in order to solve this problem. I have my attempt below. Problem: Every second, Alice can either send (bit 1) or not ...
0
votes
1answer
24 views

PRNG for compression

I'm trying to intuitively grasp information theory. You have a string of size X that contains a lot of information, say it's a movie. You have a string of size N << X which is going to be the ...
0
votes
1answer
48 views

Conditional entropy and independent conditioning variables

Let $X,Y,Z,Y',Z'$ be random variables where $Y\sim Y', Z\sim Z'$, $Y$ and $Z$ are independent, while $Y'$ and $Z'$ are, in the sense that we have $p(X,Y,Z)=p(X|Y,Z)p(Y)p(Z)$ $p(X,Y',Z')=p(X|Y',Z')p(...
1
vote
0answers
45 views

mutual information and data processing inequality for $X\to Y\to Z$ where $Y=f(X)$

Let $X\to Y\to Z$ be three random variables. The data processing inequality states $I(X;Y)\geq I(X;Z)$. Further assume $Y=f(X)$ where $f:\mathcal{X}\to\mathcal{Y}$ is an arbitrary function. What ...
2
votes
2answers
44 views

chain rule conditional entropy

I have to prove the chain-rule for conditional entropy. I kept getting stuck on one step, so I looked up a proof and found this: \begin{align}H(Y\mid X)&= \sum_{x\in\mathcal X, y\in\mathcal Y}p(x,...
4
votes
1answer
61 views

References for information theoretic statistical tools

Strange statistical concepts like spaces of probability distributions, "metrics" like Fisher information or relative entropy, and convergence with respect to these quantities are necessitated in my ...
3
votes
0answers
67 views

Differential entropy of the product of Gaussian random variables

Given two independent Gaussian random variables $X \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $Y \sim \mathcal{N}(\mu_y,\sigma_y^2)$. We look at the product distribution of these two random variables $Z=...
1
vote
2answers
79 views

Geometric distribution achieves maximum entropy for given mean

Let $X$ be a random variable with geometric distribution, ie $P(X=k)=p(1-p)^k$. If I calculated it correctly, $X$ has mean $E(X)=\frac{1-p}p$ and entropy $H(X)=-\log p - \frac{1-p}p\log{(1-p)}$ (...
0
votes
0answers
29 views

Books for large deviations and maximum entropy principle?

What are some good introductory resources for learning more about the theory of large deviations and the maximum entropy principle?
2
votes
1answer
45 views

higher moments of entropy… does the variance of $log x$ have any operational meaning?

The Shannon entropy is the average of the negative log of a list of probabilities $\{ x_1 , \dots , x_d\}$, i.e. $$H(x)= -\sum\limits_{i=1}^d x_i \log x_i$$ there are of course lots of nice ...
0
votes
0answers
33 views

Is there any definition of entropy of a stochastic process?

Entropy of finite random variables is defined in Wiki https://en.wikipedia.org/wiki/Entropy_(information_theory) Entropy rate of a stochastic process is defined in Wiki https://en.wikipedia.org/wiki/...
0
votes
1answer
52 views

What does $\operatorname{bb}()$ mean in information theory?

I'm coming across a lot of formulas in the textbook that use "bb()" however, I don't know what bb is, it isn't mentioned anywhere. For example max entropy in a binary source: $$\mathbb{H}_{max} = \...
3
votes
1answer
35 views

Relative entropy for wiener measure/wiener measure with girsanov change of drift

I've read an article on relative entropy properties that gives a result for the relative entropy of two equivalent measures as they are found in applications of girsanovs theorem. For two measures P, ...
0
votes
0answers
29 views

Comparison between Shannon's and Blackwell's measure of informativeness

I want to compare the concept of ``precision of information'' between signals $x \in X$ and states $\omega \in \Omega$ defined by Blackwell and Shannon. Denote the conditional probability ...
0
votes
0answers
29 views

Proof that the q-Gaussian is derived by maximizing Tsallis entropy

So I have to write a report paper for a course and I would like to prove that the q-Gaussian is the distribution which arises once one maximizes the Tsallis entropy. But I face difficulties in ...
1
vote
1answer
55 views

Is this some entropy I haven't heard of?

For a discrete finite probability distribution $p(s)$ the function $$\sum\limits_s p(s)\log ^2 p(s)$$ looks like the Shannon entropy but has a square on the $\log$. Is there a name for this? Or it is ...
2
votes
1answer
162 views

What is the entropy correction term for a multivariate distribution?

I wonder if anyone would be able to help me with a confusion I have got myself into please. Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a ...
1
vote
0answers
35 views

Multivariate Gaussian with singular covariance matrix

The entropy of a multivariate Gaussian is given at https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Entropy as $$\frac{1}{2}\ln((2\pi e)^n |\Sigma|).$$ Here $n$ is the dimension of the ...
3
votes
2answers
156 views

Convergence to normal distribution

Consider the probability distribution of the simple symmetric walk. That is the random variable $X_i$ equals $c$ or $-c$ with equal probability and all $X_i$ are independent and $c\geq1$. We are ...
0
votes
0answers
29 views

Approximating the set of 2-typical sequences

I have been trying to find an upper bound for the set of 2-typical sequences; here is how far I got - I would appreciate any further help very much: Let $x^n=x_1,x_2,\ldots, x_n$ be a sequence from a ...
1
vote
0answers
43 views

Entropy and the probability to guess

Let $X$ be a discrete random variable and suppose that we choose a random value $X=x_1$. Let $A$ be an event such that $H[ X \mid A] = k$, where $$H[X \mid A] = - \sum_{x} P[X =x \mid A] \log_2( P[X ...
8
votes
1answer
135 views

Entropy of matrix vector product

Consider a random $n$ by $n$ matrix $A$ whose entries are chosen from $\{0,1\}$ and a random $n$ dimensional vector $x$ whose entries are also chosen from $\{0,1\}$. Assume $n$ is large. What is ...
0
votes
1answer
50 views

Entropy of a character in a String

Taking into account the Shannon entropy, I was wondering that, if we have a String like $1122344444455$ , is this possible to find out the entropy of digit $4$ in this String? In other words, I would ...
5
votes
1answer
65 views

Maximum entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only non-negative integer entries less than or equal to $n$ ...
2
votes
1answer
28 views

Entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only positive integer entries in some finite range. I am ...
1
vote
0answers
34 views

How to show the inf can be achieved by some nonnegative $u\in H^1(M)$?

When I read some about Perelman's $\mathcal W$ function, I get stuck with the red line in the picture below.Seemly, I should to read the 8.2 Existence of minimizers of Evans' PDE. But I am not sure , ...
2
votes
1answer
45 views

Compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$

The question is to compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$. First anyone knows what this distribution is? $x$ can only take non-...
0
votes
0answers
36 views

Compute the entropy of Cauchy distribution

Compute the entropy of the density function $\dfrac{b}{{\pi ({b^2} + {x^2})}}$. I think the entropy of a density function $f(x)$ is given by $H = -\int f(x) \ln f(x) ~dx$ My calculation is $$H = - \...
1
vote
1answer
62 views

Mutual information for a continuous uniform distribution

I'm trying to compute using matlab the mutual information for an $ \infty $-PAM input (the limit of a very dense PAM constellation) for a range of snr and I got stuck. I'm working with a real-valued ...
0
votes
0answers
22 views

Lower bound on conditional entropy over multiple random variables

I am trying to compute the best subset of features for a given random variable $X_i$ from the set of given $n$ random variables. For that I am using conditional entropy to determine the best subset, ...
1
vote
0answers
29 views

Show entropy bound unit simplex

Let $\mathcal{S}_d$ be the $d$-dimensional unit simplex. Then for the norm $||x||_1 = \sum_i |x_i|$ and $0 < \varepsilon \leq 1$, $$N(\varepsilon, \mathcal{S}_d, ||\cdot||_1) \leq \left(\frac{5}{\...
1
vote
0answers
22 views

Efficient numerical calculation of Entropy of Poisson RV

There doesn't exist a closed-form solution of following infinite series (entropy of Poisson RV): $$ \displaystyle \sum_{x=0}^{\infty} \Big( \frac{e^\lambda \lambda^x}{x!} \Big) \cdot \text{log}_2\Big( ...
3
votes
1answer
108 views

Special functions related to $\sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$

While doing some caculation related to von Neumann entropy, I encountered this kind of convergent series. $$\text{Exl}(x) \equiv \sum _{n=1}^{\infty } \frac{x^n \log (n!)}{n!}$$ In my calculation, ...