# Tag Info

## Hot answers tagged information-theory

3

It's important to distinguish $H(X \mid Y)$ from $H(X \mid Y=y)$. The first is a number, the second is a function of $y$. The first is the average of the second -averaged over $p(y)$: $$H(X \mid Y) = E_y \left[H(X \mid Y=y)\right]$$ Hence, beacuse it's an average, in general we'll have $H(X \mid Y=y)>H(X \mid Y)$ for some values of $y$ and $H(X \mid ... 3 The problem is that the differential entropy$h(X)$is not a true entropy$H(X)$. For example, the diferential entropy of an uniform continuous variable in some real interval is finite (positive, or zero or even negative!), but the amount of information it provides (its "true entropy") is infinite. In particular, it's true that$I(X;Y) = h(X) - h(X|Y)$... 3 Fisher information is related to the asymptotic variability of a maximum likelihood estimator. The idea being that higher Fisher Information is associated with lower estimation error. Shannon Information is totally different, and refers to the content of the message or distribution, not its variability. Higher entropy distributions are assumed to convey ... 1 The error is in stating$I(X;Y) = h(X)$, in fact$I(X;Y)$isn't defined when$X$is a continuous random variable and$Y=X$because the random (vector) variable$(X,Y)$isn't continuous (doesn't have a joint pdf). The joint cdf$F(x,y) = P(X \leq x, Y \leq y) = P(X \leq \min(x,y))$has a "ridge" along$x=y\$ which prevents it from being differentiable. ...

Only top voted, non community-wiki answers of a minimum length are eligible