# Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

74 views

### Cross-entropy minimization - equivalent unconstrained optimization problem

I'm looking at this paper "An Alternative Method for Estimating and Simulating Maximum Entropy Densities" (https://editorialexpress.com/cgi-bin/conference/download.cgi?db_name=FEMES09&paper_id=780)...
41 views

### Proof of a fact about mutual information and entropy

It is needed to prove that for distribution (a, b, c) such that $I(a;b|c) = I(a;c|b) = I(b;c|a) = 0$ exists a random variable $d$ such that $H(d) = I(a, b, c)$ and $a,b,c$ are independent with ...
56 views

### Proof of inequality with entropy

I can not prove this inequality $2H(a,b,c) \leq H(a,b) + H(a,c) + H(b,c|a),\ H-entropy$. I tried do it by using chain rule and this inequality $H(X|Y) \leq H(X;Y)$ but without any success. Please ...
55 views

### Questions Regarding Mutual Information

I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties. I've created some MATLAB code that basically makes two ...
72 views

### How do I evaluate $\mathbb E(X\log(X))$ if $X$ has a binomial distribution, for large $n$ values?

$X\sim\mathcal {Bin}(n,p)$ I want to evaluate $\sum\limits_{x=0}^n {^n\mathrm C_x} p^x(1-p)^{n-x}x\log(x)$. Is there any way to avoid the sum because my $n$ can be very large (around $10^6$)?
48 views

### a question about entropy of run length coding

I'm doing an exercise from chapter two of $\textit {elements of information theory}$. Here is the problem and its solution, . I'm not very clear about the equation 2.36 or say why does the equation ...
926 views

### Entropy for three random variables [duplicate]

I'm just working through some information theory and entropy, and I've come into a bit of a problem. In many texts, it's easy to find the "chain rule" for entropy in two variables, and the "...
40 views

### Required bits to communicate a partial order?

Suppose that you have a ranking (i.e. a strict complete partial order) over $n$ different objects, so that the objects can be ordered as $a>b>\cdots>n$. You want to communicate the exact ...
137 views

### What is the maximum entropy distribution over all integers (ie. including negative ones) with fixed mean and variance?

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions. However, I cannot find conclusive information about what are the maximum ...
48 views

75 views

### How to calculate the mutual information between two outputs of Rayleigh fading channels

We have the two channels: $$X_{a,i} = H_{i}s_{i} +N_{a,i} \\ X_{b,i} = H_{i}s_{i} +N_{b,i}$$ for $1 \leq i \leq n$, where $H_i$ denotes the i.i.d. channel coefficient and is a zero-mean complex ...
33 views

### Relative Entropy decomposition reference

may I ask for some reference pointers? My bad as I got a classic case of losing my reference and thus unsure what I wrote was right or wrong. I tried looking my old references and internet and didn't ...
50 views

### Proving certain aspects of Entropy

I am trying to prove three properties of entropy. $1)$ $H(X|Y,Z)\le H(X|Y)$ $2)$ $H(X|Y,Z)\le H(X,Y)$ $3)$ $H(X,Y,Z)+H(Y)\le H(X,Y)+H(Y,Z)$ I have proved the third one, but it is based on part 1. ...
53 views

### Motivation for the binary entropy function

What is the motivation for the definition of the binary entropy function $H(x) = -p\log_2(p) - (1-p)\log_2(1-p)$? I understand that we want the entropy to be zero at $p = 0$ and $p = 1$ (no randomness)...
120 views

### Covering Number of Lipschitz Function Space

On page 12 of the slide deck here, the author gives an example where a lower bound (and upper bound, but I am particularly interested in the lower bound) on the $\epsilon$-covering number of a ...
71 views

### Limit of $\frac{1}{n}\log({n\choose np})$ without using Stirling's formula

I am trying to evaluate the following limit: $$\forall p\in(0,1) ,\lim_{n\rightarrow \infty} \frac{1}{n}\log{n\choose \lfloor np \rfloor} =H(p),$$ where $\lfloor x\rfloor$ means the greatest integer ...
62 views

193 views

### Entropy of floating number array

I am familiar with shanon's definition of entropy. $$H(P) = - \sum_{i=1}^n p_i \cdot \log_2(\mathcal p_i)$$ I am today in the situation that I'd like to compute an entropy like function but for a ...
64 views

### Entropy Solution of $u_t+(u^2/2)_x=0$

Given the initial data $$g(x)= \cases{ 1 & x< -1 \\ 0 & -1 < x< 0 \\ 2 & 0 < x< 1 \\ 0 & 1 < x \\ }$$ What is the entropy solution of $u_t+(u^2/2)_x=0$?
82 views

### Reference for entropy of the binomial distribution?

The Wikipedia page Binomial distribution says that the entropy of the Binomial(n,p) is $\frac{1}{2}\log_2\left(2\pi e n p (1-p)\right) + O\left(\frac{1}{n}\right)$. What is a reference (paper or ...
We have the transformation $T: [0,1) \rightarrow [0,1)$ given by $Tx = \beta x \text{ mod } 1$ with $\beta = \frac{1+ \sqrt{5}}{2}$. Calculate the entropy $h_{\mu}(T)$ of $T$ wrt the invariant ...
I have the following problem: I want to find the probability density $p$ which maximizes the Shannon entropy $$S := - \int_{x_b}^{x_c} dx ~ p(x) \log (p(x))$$ under the ...