# Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

38 views

23 views

61 views

37 views

### mutual information entropy problem

In mutual information we have: if $x$ and $y$ are independent then $p(x,y)=p(x)p(y)$ and then $I(X;Y)=0$. Do If $I (X;Y) = 0$ when $x$ and $y$ are not necessarily independent?
33 views

The entropy of a password of a fixed length $n$ and $c$ possible characters is calculated by $n*log_2(c)= log_2(c^n)$, see also here: https://ritcyberselfdefense.wordpress.com/2011/09/24/how-to-...
169 views

### Is entropy of prime numbers smaller?

Seems that entropy (in information theory) can be expressed as a measure of how unpredictable is each bit of information. I have done a little experiment: I've measured entropy of the binary ...
50 views

### Conditional joint entropy of two random variables

I am trying to prove the formula that gives the joint entropy of the random variables $X$ and $Y$ given $Z$ which is: $$H(X,Y|Z) = H(X|Z) + H(Y|X,Z)$$ based on the definition of conditional entropy ...
78 views

### Entropy of union of multisets

Assigning a random variable to some multiset: Assume that $S$ is a multiset. We can think of $S$ as independent sampling from some random variable. For instance, $S = \{H, H, T, T, T\}$ can be thought ...
38 views

### How can I compare the entropy of two vectors with different sum?

I wanna compare the entropy of two vectors, e.g. [1,2,3] vs [0.2, 0.1, 0.3]. Note that the two vectors have different sum, i.e. 6 and 0.6. Basically I wanna know which vector is more unbalanced. ...
189 views

### Jensen's inequality proof

The standard proof for Jensen's inequality using taylor expansion around a point $x_0$ involves using only first 3 terms of the Taylor series till $f^{\prime \prime}(x)$. Why are we able to ignore the ...
300 views

### An upper bound of binary entropy

Binary entropy is given by $$H_{\mathrm b}(p) = -p \log_2 p - (1 - p) \log_2 (1 - p), \hspace{6 mm} p \le \frac{1}{2}$$ How can I prove that $$H_{\mathrm b}(p) \le 2 \sqrt{p(1-p)}$$
28 views

### Prove entropy theorem

Doing a course of cryptography I have been asked to prove the following: $H(X,Y) = H(Y) +H(X|Y)$. But I simply do not know where to start, so a hint in the right direction would be very much ...
33 views

### Open covers and $(n,\varepsilon)$-separating/ spanning sets: proving three inequalities

In Peter Walters' book An Introduction to Ergodic Theory, one can find the following corollary (p. 174 in my edition). At the end of this thread, I tried to prove it. It would be great if you could ...
26 views

### Log base change problem, Multivariate Gaussian differential entropy proof

I am working through a proof in this document http://ee.tamu.edu/~georghiades/courses/ftp647/Chapter7.pdf for Theorem 3 (The entropy of a multivariate Gaussian distribution): Let X = (X1, X2, · · ·...
42 views

### Is the topological entropy of a continuous map $T\colon X\to X$ zero if $X$ is a finite topological space?

Let $X$ be a finite topological space and $T\colon X\to X$ continuous. As the title already suggests, I am wondering if the topological entropy of $T$, denoted by $h(X,T)$, then is $0$. As far as I ...
22 views

### Source coding with 2 distinct distributions and entropies

I'm learning about source coding, and many of the books/resources I've read give examples of the source $X^n$ being defined as a sequence of iid random variables. How about when the sequence is ...
148 views

### Very special geometric shape (No name yet?)

I suppose this geometric shape is something very 'special'. I cannot clarify in short about being 'special', but I think this shape stands together with such special shapes like the square and the ...