I was reading about the definition of conditional entropy. Let $H$ be the entropy of an event, $X,Y$ be the events with outcome $x,y$. By definition, $H(X|Y ) + H(Y) = H(X, Y ) \tag{*}\label{*}$ Furthermore \begin{align} H(X,Y) &= \sum_{x,y}−p(x, y)\log p(x,y) \\ &=\sum_{x,y}−p(x, y) \log [p(y)p(x|y)] \\ &=\sum_{x,y}−p(x|y)p(y)[\log p(y)+\log p(x|y)] \tag{**}\label{**}\\ \end{align} Ideally, I want $H(X,Y)=-\sum_{x,y}p(x|y)\log p(x|y)-\sum_{y}p(y)\log p(y)$, so that it agrees with the definition, or $\eqref{*}=\eqref{**}$. However it doesn't seem to be equal to me. What mistake have I made?
2
-
1$\begingroup$ conditional entropy $H(X|Y):=-\sum_{x,y}p(x,y)\log(p(x|y))=-\sum_{x,y}p(,x,y)\log\big(\frac{p(x,y)}{p(y)}\big)$. See here. $\endgroup$– MittensCommented Dec 19, 2023 at 2:51
-
1$\begingroup$ In your calculation, you're just missing one step: $$ \sum_{x,y} p(x|y) p(y) \log p(y) = \sum_y \left( p(y) \log p(y) \sum_x p(x|y)\right),$$ but $\sum_x p(x|y) = 1$, so this is just $\sum_y p(y) \log p(y)$. $\endgroup$– stochasticboy321Commented Dec 19, 2023 at 13:56
Add a comment
|