This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

2
votes
1answer
64 views

Password entropy of famous xkcd comic

The famous xkcd comic about password strength calculates the entropy of the 11-character password "Tr0ub4dor&3" with 28 bits of entropy. When following the ASCII-95-chart, we have 95 possible ...
1
vote
1answer
34 views

Does entropy inequality hold for convex combination

I have two pairs of Random Variables, $(\mathbb{X},\mathbb{Y})$ and $(\mathbb{M},\mathbb{N})$ which satisfies, $H(\mathbb{X})>H(\mathbb{Y})$ and $H(\mathbb{M})>H(\mathbb{N})$. For some convex ...
1
vote
1answer
58 views

Proving that the entropy is zero given conditional entropies

Let's suppose we have 4 random variables $X,Y,Z$ and $T$ and that the following equations hold about the entropy: $$H(T|X)=H(T)$$ $$H(T|X,Y)=0$$ $$H(T|Y)=H(T)$$ $$H(Y|Z)=0$$ $$H(T|Z)=0$$ Also, the ...
1
vote
0answers
33 views

conditional mutual information

I have a question about mutual information $$I(Z ; T/X,Y) = I(T/X,Y ; Z)$$ $T,X,Y,Z$ are random variables is this statement accurate? if it is true and I know that I(Z;T/X,Y) = H(Z/X,Y) - H(Z/T,X,...
1
vote
1answer
68 views

Entropy and Mutual Information

Consider two discrete random variables $X$ $\{x_1,x_2,\dots,x_n\}$ and $Y$ $\{y_1,y_2,\dots, y_n\}$. Lets say that entropy $H(X)=0$ i.e. $X$ has a probability distribution s.t. $P(X=x_j) = 1$ for only ...
0
votes
1answer
58 views

Conditional entropy under quantization

Let $X$ be a continuous random variable and $X^n$ its quantization that becomes finer with larger $n$. Let $Y$ be a deterministic function of $X$. Then we have that the conditional entropy $$H(Y|X) = ...
0
votes
1answer
31 views

Not understanding steps in derivation for entropy of a Gaussian random variable

Can someone explain the last two steps in the derivation given below? This is the derivation of the entropy of a Gaussian random variable:
0
votes
1answer
35 views

If $X$ to $Y$ to $Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$

If $X \to Y \to Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$. I said the statement was true, and from $I(X;Y)\ge I(X;Z)$ by definition, thus $H(X) - H(X\mid Y) \ge H(X)-H(X\mid ...
1
vote
1answer
45 views

Rate distortion function with infinite distortion

I am working through the problems in Elements of Information Theory by Cover and Thomas and have come across the following problem I couldn't answer. The problem is to find the rate distortion ...
1
vote
0answers
27 views

Entropy of a dictionary

I have an english dictionary (a file that contains a list of words) and I want to calculate: given a path tree (a word), measure $H(C_{l+1}|C_l=c_l, C_{l-1}=c_{l-1}, ...)$ for a few levels of the ...
-1
votes
1answer
21 views

Does the Information Gain algorithm favor a high-entropy attribute or a low-entropy one?

This might not be mutual to mathematics but it does relate to Information-Theory. My question is: Does the InformationGain algorithm, in Decision-Tree machine-learning, favor a high-entropy ...
0
votes
0answers
33 views

Channels with memory have higher capacity

I am working through Elements of Information Theory by Cover and Thomas and have come across the following solution to one of their problems that I don't understand. Consider a binary, symmetric ...
1
vote
0answers
14 views

Entropy of first char of a word in dictionary

Suppose I have an English dictionary, that is a list of words in a file. I have to calculate the entropy of the first char. I calculated the probability of each first char ($P_a, P_b, \dots, P_z$) in ...
5
votes
1answer
185 views

Is there a possibility to determine/ estimate the topological entropy?

By $E$, denote the set of excited states $E=\left\{1,2,\ldots,e\right\}$ and by $R$ the set of refractory states $R=\left\{e+1,e+2,\ldots,e+r\right\}$. By $0$, denote the equilibrium state. The ...
1
vote
1answer
25 views

Entropy Inequality $H(X| g(Y))\geq H(X|Y)$

Let $X,Y$ be random discrete variables. $H(X) = -\sum\limits_{x}P\{X=x\}\operatorname{log}_2P\{X=x\}$ be the entropy-function. It is known fact that that $H(g(Y))\leq H(Y)$. I want to prove the ...
6
votes
0answers
61 views

At what rate does the entropy of shuffled cards converge?

Consider a somewhat primitive method of shuffling a stack of $n$ cards: In every step, take the top card and insert it at a uniformly randomly selected one of the $n$ possible positions above, between ...
0
votes
1answer
34 views

Existence of measure(s) of maximal entropy, given a finite-to-one chaotic global attractor $A$ which is, moreover, the non-wandering set

Let $X$ be a compact metric space and let $f\colon X\to X$ define some dynamics, $f$ being continuous and finite-to-one on a global attractor $A\subset X$. Moreover, $A$ is the non-wandering set $\...
0
votes
0answers
46 views

Is my entropy calculation correct? Clustering entropy example

I would like to calculate entropy of this example scheme http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html Equation of entropy Then the entropy is (the first line) ...
5
votes
1answer
98 views

Mutual information vs Information Gain

I always thought that mutual information and information gain refer to the same thing, however looking at Wikipedia: http://en.wikipedia.org/wiki/Information_gain https://en.wikipedia.org/wiki/...
0
votes
0answers
13 views

Is there an obvious explanation for why the entropy formula is better than standard deviation as a numerical summary for spread of a distribution?

I'm currently reading Murphy's Machine Learning book, and he briefly introduces the notion of entropy for a random variable. He states that the entropy is $$ \mathbb{H}(X)=-\int p(x)\log_{2}p(x)\,dx, ...
0
votes
1answer
32 views

Entropy of gamma-exponential compound distribution

Following this question, I have the PDF of a gamma-exponential compound distribution as $$f(y) = \frac{\alpha\beta^{\alpha}} {(y+\beta)^{(\alpha+1)}} $$ For my application I need the entropy of ...
0
votes
0answers
62 views

Generalization of Shannon's source coding theorem with a posteriori entropies

This doubt is with reference to section 5-5 of "Information theory and Coding" by Prof. Norman Abramson. Under the topic "A generalization of Shannon's First Theorem", the text discusses how knowledge ...
2
votes
0answers
22 views

Showing that for a family of subsets of $[n]$ enough elements appear in high frequencies

Let $\mathcal{F} \subseteq 2^{[n]}$ a familiy of subsets. Assume that the following applies: For every $A \subseteq [n]$ , such that $|A|\leq \alpha n$ ($\alpha > 0$ is given), there's a subset $...
1
vote
1answer
52 views

Calculating the capacity of a Binary Channel?

I'm pretty new in information theory so I need your suggestions or references in order to solve this problem. I have my attempt below. Problem: Every second, Alice can either send (bit 1) or not ...
0
votes
1answer
24 views

PRNG for compression

I'm trying to intuitively grasp information theory. You have a string of size X that contains a lot of information, say it's a movie. You have a string of size N << X which is going to be the ...
0
votes
1answer
49 views

Conditional entropy and independent conditioning variables

Let $X,Y,Z,Y',Z'$ be random variables where $Y\sim Y', Z\sim Z'$, $Y$ and $Z$ are independent, while $Y'$ and $Z'$ are, in the sense that we have $p(X,Y,Z)=p(X|Y,Z)p(Y)p(Z)$ $p(X,Y',Z')=p(X|Y',Z')p(...
1
vote
0answers
52 views

mutual information and data processing inequality for $X\to Y\to Z$ where $Y=f(X)$

Let $X\to Y\to Z$ be three random variables. The data processing inequality states $I(X;Y)\geq I(X;Z)$. Further assume $Y=f(X)$ where $f:\mathcal{X}\to\mathcal{Y}$ is an arbitrary function. What ...
2
votes
2answers
45 views

chain rule conditional entropy

I have to prove the chain-rule for conditional entropy. I kept getting stuck on one step, so I looked up a proof and found this: \begin{align}H(Y\mid X)&= \sum_{x\in\mathcal X, y\in\mathcal Y}p(x,...
4
votes
1answer
62 views

References for information theoretic statistical tools

Strange statistical concepts like spaces of probability distributions, "metrics" like Fisher information or relative entropy, and convergence with respect to these quantities are necessitated in my ...
3
votes
0answers
70 views

Differential entropy of the product of Gaussian random variables

Given two independent Gaussian random variables $X \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $Y \sim \mathcal{N}(\mu_y,\sigma_y^2)$. We look at the product distribution of these two random variables $Z=...
1
vote
2answers
83 views

Geometric distribution achieves maximum entropy for given mean

Let $X$ be a random variable with geometric distribution, ie $P(X=k)=p(1-p)^k$. If I calculated it correctly, $X$ has mean $E(X)=\frac{1-p}p$ and entropy $H(X)=-\log p - \frac{1-p}p\log{(1-p)}$ (...
0
votes
0answers
33 views

Books for large deviations and maximum entropy principle?

What are some good introductory resources for learning more about the theory of large deviations and the maximum entropy principle?
2
votes
1answer
46 views

higher moments of entropy… does the variance of $log x$ have any operational meaning?

The Shannon entropy is the average of the negative log of a list of probabilities $\{ x_1 , \dots , x_d\}$, i.e. $$H(x)= -\sum\limits_{i=1}^d x_i \log x_i$$ there are of course lots of nice ...
0
votes
0answers
34 views

Is there any definition of entropy of a stochastic process?

Entropy of finite random variables is defined in Wiki https://en.wikipedia.org/wiki/Entropy_(information_theory) Entropy rate of a stochastic process is defined in Wiki https://en.wikipedia.org/wiki/...
0
votes
1answer
53 views

What does $\operatorname{bb}()$ mean in information theory?

I'm coming across a lot of formulas in the textbook that use "bb()" however, I don't know what bb is, it isn't mentioned anywhere. For example max entropy in a binary source: $$\mathbb{H}_{max} = \...
3
votes
1answer
37 views

Relative entropy for wiener measure/wiener measure with girsanov change of drift

I've read an article on relative entropy properties that gives a result for the relative entropy of two equivalent measures as they are found in applications of girsanovs theorem. For two measures P, ...
0
votes
0answers
31 views

Comparison between Shannon's and Blackwell's measure of informativeness

I want to compare the concept of ``precision of information'' between signals $x \in X$ and states $\omega \in \Omega$ defined by Blackwell and Shannon. Denote the conditional probability ...
0
votes
0answers
29 views

Proof that the q-Gaussian is derived by maximizing Tsallis entropy

So I have to write a report paper for a course and I would like to prove that the q-Gaussian is the distribution which arises once one maximizes the Tsallis entropy. But I face difficulties in ...
1
vote
1answer
55 views

Is this some entropy I haven't heard of?

For a discrete finite probability distribution $p(s)$ the function $$\sum\limits_s p(s)\log ^2 p(s)$$ looks like the Shannon entropy but has a square on the $\log$. Is there a name for this? Or it is ...
2
votes
1answer
163 views

What is the entropy correction term for a multivariate distribution?

I wonder if anyone would be able to help me with a confusion I have got myself into please. Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a ...
1
vote
0answers
38 views

Multivariate Gaussian with singular covariance matrix

The entropy of a multivariate Gaussian is given at https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Entropy as $$\frac{1}{2}\ln((2\pi e)^n |\Sigma|).$$ Here $n$ is the dimension of the ...
3
votes
2answers
157 views

Convergence to normal distribution

Consider the probability distribution of the simple symmetric walk. That is the random variable $X_i$ equals $c$ or $-c$ with equal probability and all $X_i$ are independent and $c\geq1$. We are ...
0
votes
0answers
29 views

Approximating the set of 2-typical sequences

I have been trying to find an upper bound for the set of 2-typical sequences; here is how far I got - I would appreciate any further help very much: Let $x^n=x_1,x_2,\ldots, x_n$ be a sequence from a ...
1
vote
0answers
46 views

Entropy and the probability to guess

Let $X$ be a discrete random variable and suppose that we choose a random value $X=x_1$. Let $A$ be an event such that $H[ X \mid A] = k$, where $$H[X \mid A] = - \sum_{x} P[X =x \mid A] \log_2( P[X ...
8
votes
1answer
135 views

Entropy of matrix vector product

Consider a random $n$ by $n$ matrix $A$ whose entries are chosen from $\{0,1\}$ and a random $n$ dimensional vector $x$ whose entries are also chosen from $\{0,1\}$. Assume $n$ is large. What is ...
0
votes
1answer
51 views

Entropy of a character in a String

Taking into account the Shannon entropy, I was wondering that, if we have a String like $1122344444455$ , is this possible to find out the entropy of digit $4$ in this String? In other words, I would ...
5
votes
1answer
65 views

Maximum entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only non-negative integer entries less than or equal to $n$ ...
2
votes
1answer
28 views

Entropy of the sum of two vectors

Consider two identically distributed independent random vectors $X$ and $Y$ of dimension $n$ (assume $n$ is large). Both $X$ and $Y$ have only positive integer entries in some finite range. I am ...
1
vote
0answers
34 views

How to show the inf can be achieved by some nonnegative $u\in H^1(M)$?

When I read some about Perelman's $\mathcal W$ function, I get stuck with the red line in the picture below.Seemly, I should to read the 8.2 Existence of minimizers of Evans' PDE. But I am not sure , ...
2
votes
1answer
45 views

Compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$

The question is to compute the entropy of density function $\frac{{x{e^{ - \frac{{{x^2}}}{{2t}}}}}}{{\sqrt {2\pi {t^3}} }}$. First anyone knows what this distribution is? $x$ can only take non-...