Consider two system $X$ and $Y$ described by probabilities distribution.

We define the conditionnal entropy of $X$ knowing $Y$ as :

$$S_{X|Y}=\sum_y p(y) \left( - \sum_{x} p(x|y) \log(p(x|y)) \right) $$

If I understood well, this quantity is supposed to quantify the entropy of $X$ once I know my system $Y$.

In a way I understand the definition, $-\sum_{x} p(x|y) \log(p(x|y))$ is the entropy of $X$ once I know for sure the state in which $Y$ is : $y$.

Then, I do the average on all the possible states of $Y$ doing the : $\sum_y p(y)$.

So things are confused in my mind : $S_{X|Y}$ quantifies the lack of information on $X$ once we know $Y$. But in the same time, we don't know $Y$.

I could say that this quantity thus quantify the lack of information on $X$ once we know $Y$ (which is lower than the lack of information without knowing $Y$). But as we don't know $Y$ we do the average on all possible outcomes of $Y$.

But for me it doesn't really mean anything to say this : we know $Y$ or we don't. This is why I am a little lost.

I would like to have a clear explanation of what is happening here, I think I miss a step in the reasoning. What does really mean the quantity $S_{X|Y}$.

  • $\begingroup$ ``we know $Y$ or we don't''. To be accurate, we only know the realization of $Y$. It therefore makes sense to consider the average entropy (over all possible realizations of $Y$), the same way it makes sense to consider the expectation of some function of a random variable as a quantity that provides useful and intuitive information. $\endgroup$ – Stelios Feb 10 at 21:59
  • $\begingroup$ @Stelios Actually I am not sure to really understand the point in the end. Can we say that the conditionnal entropy is the lack of knowledge I have on $X$ knowing the probability of realistions of $Y$ and the correlations between $X$ and $Y$ ? $\endgroup$ – StarBucK Apr 22 at 19:14
  • $\begingroup$ Your search for a "physical" interpretation to $H(X|Y)$ is similar to the following question: If $Z$ is the outcome of a fair dice (i.e., $Z \in \{1,2,\ldots,6 \}$ with equal probability), what is the meaning of its mean value equal to $\mathbb{E}(Z)=3.5$? The answer to the latter is that there is no physical meaning: It is just the average of the realizations of the outcome $Z$. Similarly, $H(X|Y)$ is just the average of the entropies $H(X|Y=y)$ (over the realizations of $Y$). Nothing more than that. $\endgroup$ – Stelios Apr 22 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.