# Conditionnal entropy : intuitive interpretation

Consider two system $$X$$ and $$Y$$ described by probabilities distribution.

We define the conditionnal entropy of $$X$$ knowing $$Y$$ as :

$$S_{X|Y}=\sum_y p(y) \left( - \sum_{x} p(x|y) \log(p(x|y)) \right)$$

If I understood well, this quantity is supposed to quantify the entropy of $$X$$ once I know my system $$Y$$.

In a way I understand the definition, $$-\sum_{x} p(x|y) \log(p(x|y))$$ is the entropy of $$X$$ once I know for sure the state in which $$Y$$ is : $$y$$.

Then, I do the average on all the possible states of $$Y$$ doing the : $$\sum_y p(y)$$.

So things are confused in my mind : $$S_{X|Y}$$ quantifies the lack of information on $$X$$ once we know $$Y$$. But in the same time, we don't know $$Y$$.

I could say that this quantity thus quantify the lack of information on $$X$$ once we know $$Y$$ (which is lower than the lack of information without knowing $$Y$$). But as we don't know $$Y$$ we do the average on all possible outcomes of $$Y$$.

But for me it doesn't really mean anything to say this : we know $$Y$$ or we don't. This is why I am a little lost.

I would like to have a clear explanation of what is happening here, I think I miss a step in the reasoning. What does really mean the quantity $$S_{X|Y}$$.

• we know $Y$ or we don't''. To be accurate, we only know the realization of $Y$. It therefore makes sense to consider the average entropy (over all possible realizations of $Y$), the same way it makes sense to consider the expectation of some function of a random variable as a quantity that provides useful and intuitive information. – Stelios Feb 10 at 21:59
• @Stelios Actually I am not sure to really understand the point in the end. Can we say that the conditionnal entropy is the lack of knowledge I have on $X$ knowing the probability of realistions of $Y$ and the correlations between $X$ and $Y$ ? – StarBucK Apr 22 at 19:14
• Your search for a "physical" interpretation to $H(X|Y)$ is similar to the following question: If $Z$ is the outcome of a fair dice (i.e., $Z \in \{1,2,\ldots,6 \}$ with equal probability), what is the meaning of its mean value equal to $\mathbb{E}(Z)=3.5$? The answer to the latter is that there is no physical meaning: It is just the average of the realizations of the outcome $Z$. Similarly, $H(X|Y)$ is just the average of the entropies $H(X|Y=y)$ (over the realizations of $Y$). Nothing more than that. – Stelios Apr 22 at 20:31