Tag Info

New answers tagged

1

Your calculation is wrong. The correct answer for four digits is $$1.7820 .$$ The correct interpretation of the entropy is as follows. Assume that you have to code an infinite sequence of independent random variables distributed identically; taking different values (symbols) with the given probabilities. Then create disjoint sequence of length $N$ out of ...


0

This is basically Fano's inequality. Sketch: Let $x$ be a random variable such that $x=1$ iff $a=b$, $x=0$ otherwise. Then use the chain rule (conditioned on $b$): $$H(a,x|b)=H(a|x,b) + H(x|b) = H(x|a,b)+H(a|b)$$ Then note that $H(x|a,b)=0$, and $H(x|b)\le H(x)\le 1$. Furthermore, $H(a|x,b)=P(x=0) H(a|x=0,b) + P(x=1)H(a|x=1,b)$ with $H(a|x=1,b)=0$ and ...


0

$I(a;b) = \frac{3h}{4}$ and $I(a;c) = \frac{3h}{4}$. Hence, $I(a;b;c) \ge \frac{h}{2}$. So if $I(b;c) \ne I(a;b;c)$ then $I(b;c) > \frac{h}{2}$ else $I(b;c) \ge \frac{h}{2}$.


-2

If function $G \in L^1(R^d)$ we can perform Fourier transfer of divergence. This end up in: $$0 = \digamma[\nabla(\nabla G + xG)] = i<k \cdot \hat{F}>$$ where $\hat{F} = ik\hat{G} +i\nabla\hat{G}$ and $\hat{G} = \digamma[G]$ Now, $\hat{F}$ vector is parallel to $k$, i.e. we do not have any perpendicular components that vanish in scalar product $<k ...


1

Hand calculation is not that painful. Fortunately Wikipedia gives the result we are aiming for. The density of the truncated normal over the interval $[a,b]$ is $$f(x)=\frac1{\sigma Z}\phi\left({x-\mu\over\sigma}\right)$$ where $\phi$ is the standard normal density and $Z$ is the normalizing constant $$Z:=\int_\alpha^\beta\phi(t)\,dt\;, $$ where the limits ...


2

Why try to perform this by hand, rather than symbolically by computer? $\int\limits_a^b N(\mu, \sigma; x) {\rm ln}(N(\mu, \sigma; x)) dx =$ $ \frac{\sqrt{\frac{2}{\pi }} \left(b e^{-\frac{b^2}{2}}-a e^{-\frac{a^2}{2}}\right)+(1+\log (2)+\log (\pi )) \text{erf}\left(\frac{a}{\sqrt{2}}\right)-(1+\log (2)+\log (\pi )) ...


1

To compute $h(U_m)$, you can consider the corresponding the shift on $Y\subset(\{0,1,2\}^{\mathbb{Z}})^{\mathbb{N}}$ which $Y=\{(x,Tx,T^2x,\ldots):x\in X\}$. You can see that $(X,T)$ is conjugated to $(Y,\sigma)$ where $\sigma$ is the left shift. Note that the partition $U_m$ is corresponding to the partition of $Y$ into sets of the form ...


1

Now, my intuition was telling me that, because X and Y are roughly the same, then $I(X;Y)=I(Y;X)≈H(X)≈H(Y)$ That's wrong. If $X$ and $Y$ are independent (as it seems here), then $$I(X;Y)=H(X)-H(X|Y) = H(X)-H(X)=0$$ (which is consistent with the intuitive notion of mutual information: if $Y$ is independent from $X$, then $Y$ gives me no information about ...


0

$$H(a,b,c) = H(a)+H(b,c\mid a) \tag{1}$$ $$H(a,b) = H(a) + H(b\mid a) \tag{2}$$ $$H(a,c) = H(a) + H(c\mid a) \tag{3}$$ Hence $$2H(a,b,c) = 2 H(a) + 2H(b\mid c,a)=\\=H(a,b)+H(a,c) + H(b,c\mid a) + H(b,c\mid a)- H(b|a)-H(c\mid a) \tag{4}$$ Then we need to show that $$ H(b,c\mid a)- H(b\mid a)-H(c\mid a) \le 0$$ or $$ H(b,c\mid a) \le H(b\mid ...


3

Under the assumption* that if $X = 0$, $X \log X = 0$, the usual approach is to use the delta method with the choice $g(x) = x \log x$, so that $$\operatorname{E}[g(X)] \approx g(\operatorname{E}[X]) + \frac{1}{2}g''(\operatorname{E}[X]) \operatorname{Var}[X].$$ This gives in your case $$\operatorname{E}[X \log X] \approx np \log np + \frac{np(1-p)}{2np} = ...


2

The distribution of a binomial is strongly concentrated around $np$, so under reasonable assumptions, the answer would be roughly $np \log (np)$. For example, when $n=1000$ and $p=1/2$, the true answer is 3107.55, while this approximation gives 3107.30.


0

If ${\bf X}$ and ${\bf Y}$ are related by a one-to-one function, then $H({\bf X})=H({\bf Y})$. This is a fundamental property, which is also intuitively obvious. It can be proved in several ways, one is using exercise 2.4 of the same book : $H(g({\bf X})) \le H({\bf X})$ But the variables $(X_1,X_2 \cdots X_n)$ and $(X_i,{\bf R})$ (for any given $i$) are ...


2

Exactly as ther other answer mentions, one can expand (recursively) the joint entropy of $n$ variables to the joint and conditional entropy of $n-1$ variables and so on. Similarly to the way the joint probability of $n$ variables can be reduced to the computation of the joint and conditional probability of $n-1$ variables (by the definition of conditional ...


2

You can combine the "conditional chain rule" and the "chain rule" to extend the joint entropy from two to three variables in a variety of ways, as follows:- $$\begin{align}H(X,Y,Z) &= H(X|Y,Z) + \color{blue}{H(Y,Z)}\\&=\color{red}{H(X|Y,Z)} + \color{blue}{H(Y|Z)+H(Z)}\\&=\color{red}{H(X,Y|Z)-H(Y|Z)}+H(Y|Z)+H(Z)\\&=H(X,Y|Z)+H(Z)\end{align}$$



Top 50 recent answers are included