Tag Info

Hot answers tagged

5

$n=2$ is sufficiently large. We then have $$\begin{align} P(00) &= 9/16 \\ P(01) &= 3/16 \\ P(10) &= 3/16 \\ P(11) &= 1/16 \end{align} $$ And so we can construct a Huffman code: 00 coded as 0 10 coded as 10 01 coded as 110 11 coded as 111 The expected cost of sending two coin flips is now $$ 1 \cdot \frac9{16} + 2 \cdot \frac 3{16} + 3 ...


2

Why did we define the requirements for self-information as above? Just because they seem reasonable, they fit with the concept we are trying to formalize. The first one says that an event with low probability give us high information. The second one says that the information a pair of two independent events is the sum of the individual informations. ...


2

I assume that you have diggested the proof of the Channel Coding Theorem (7.7 ; esp. p. 203). This is analogous, just a bit more complex. In the example, the "good" set of inputs and outputs is the tuple $(\mathbf{X_1}(1), \mathbf{X_2}(1), \mathbf{Y})$, and (in the case of your doubt) we are interested in the event that some "bad" tuple of the form ...


2

The mutual information tells you nothing about the entropy of C. The mutual information can be $l$ and the entropy of $C$ can be $0$: set $$ A(i) = 1-B(i) = \begin{cases} 0 \mbox{ with probability } 1/2, \\ 1 \mbox{ with probability } 1/2. \end{cases}$$ The mutual information can be $l$ and the entropy of $C$ can be $l$: set $$A(i) = B(i) = \begin{cases} 0 ...


1

The greatest lower bound is 0 bits: $A=1-B$, so $I(A,B)=H(A)=H(B)=0.5$. But with your definition of $f$, $C=0$, and hence $H(C)=0$ An upper bound is given in the comments by Peter Shor.


1

It depends on $W$. For example if $W=f(Y)$, then you are right, however if W is independent of $Y$, then conditioning on $W$ can increase the conditional mutual information $I(X;Z|YW)$! For example if $W$ is $mod 2$ sum of $X,Z$ which is independent of $X,Z$ then $I(X;Z|Y)=0, I(X;Z|YW)=1$ which violates the desired Markov chain.


1

Hint: Let $Z_a=X+aY$, and $w(a)=h(Z_a)=h(X+aY)$, in nats. Then show that $w(a)$ increases, using de Bruijnā€™s identity : $$ \frac{\partial }{\partial t} h(X+\sqrt{t}\,Y)=\frac{1}{2}J(X+\sqrt{t}\,Y)\ge 0$$ where $J(\cdot)$ is the Fisher information (see eg Cover & Thomas, Theorem 17.7.2).


1

Your intuition is correct but your calculation of entropy is not, the differential entropy of the uniform distribution on $(0,\frac{1}{2^n})$ is $$-2^n\int_0^{\frac{1}{2^n}}\log_22^ndx=-n$$ The concept of entropy for continuous random variables (differential entropy) is not well defined, since a real number requires a theoretically infinite number of bits ...


1

Let n be a large even number. Take n equally spaced points along the unit interval. A probability distribution which gives equal weight to all these points converges weakly to the uniform distribution. The product of two copies of this distribution converges weakly to uniform on the unit square. Since this distribution is equal to the product of it's ...


1

The mutual information of two normal variables with correlation factor $\rho$ is $$ I(X;Y) = -\frac{1}{2} \log(1-\rho^2) \tag{1}$$ See proof in any Information Theory textbook - eg. In our case, letting $X=A+B$ and $Y=A+C$, we have $\sigma_X^2 = \sigma_A^2+\sigma_B^2$, $\sigma_Y^2 = \sigma_A^2+\sigma_C^2$, $\sigma_{XY}=E(X Y) = \sigma_A^2$. Hence $$ ...



Only top voted, non community-wiki answers of a minimum length are eligible