In Cover and Thomas' "Elements of Information Theory", the joint entropy $H(X,Y)$ is defined, but they state that this definition is nothing new if we consider that it is the entropy of a single vector valued random variable $(X,Y)$. Then it goes on to define conditional entropy by itself, but the previous remark got me thinking if this too could be just the entropy of a random variable $X|Y$.

But is $X|Y$ really a random variable? It seems that much of the time we talk about random variables, we use them to state facts about the probability distributions associated with them, which is interesting because (correct me if I am wrong) formally they are just functions from $\Omega$ to $\mathbb{R}^n$ and they don't carry information about the distribution. Entropy of a random variable is actually entropy of a PMF which we associate with that random variable in our heads.

That leads me to beleive that $X|Y = X$ formally, in the sense of functions, and the only distinction is that we "change the PMF of $X$". Am I making any sense or is this interpretation wrong?

If I am correct, an additional question would be, why is it common practice to stretch the definitions in this way and talk about RVs so freely when the actual object of interest is the distribution?

  • $\begingroup$ Aren't you confusing $H(X|Y)$ with $H(X|Y=y)$?. Notice that the difference of those two things is NOT analogous to the difference between (say) $E(X|Y)$ and $E(X|Y=y)$ $\endgroup$ – leonbloy Apr 29 '14 at 15:48

$X|Y$ is not a RV, but $E(X|Y)$ is!

And you can define $H(X|Y) = H(X) - H(E(X|Y))$. For example, if $X,Y$ are independent, then $E(X|Y)$ is just a number, hence entropy 0, and then $H(X|Y) = H(X)$.

"formally they are just functions from $\Omega$ to $R^n$ and they don't carry information about the distribution." not completely correct, because $\Omega$ has to be endowed with a measure. In fact, the exact $\Omega$ does not really matter as the same RV can be defined in many possible ways on many sample spaces.

What $E(X|Y)$ does, is reduces "granularity" of subsets of $\Omega$. For example, for RV $X$, $\omega_1$ and $\omega_2$ might lead to different values, but for $E(X|Y)$, to the same value.

  • $\begingroup$ Can you explain why X|Y isn't a random variable? Is it a technical problem in measure theory that stops the "intuitive" interpretation from working? $\endgroup$ – nomen Apr 29 '14 at 15:42
  • $\begingroup$ The expected value of an RV is another instance of the same issue, because, although not explicitely stated, it depends on the distribution of an RV, and strictly speaking RVs dont carry their distributions with them. I know the issue is rather silly, but I just want to check that I am not making a mistake in saying this. $\endgroup$ – spelufo Apr 29 '14 at 15:47
  • $\begingroup$ to clarify, $X$ is a mapping, but the distribution of $X$ has to do with both the mapping and the measure on $\Omega$. Can you define $X|Y$? How? If $X|Y = X$ then it's the same RV. You're not allowed to change $\Omega$ mid-stride. But with $E(X|Y)$, you just change the mapping. $\endgroup$ – PA6OTA Apr 29 '14 at 15:52
  • $\begingroup$ OK, but the information about the distribution is not contained in the random variable object, is it? It's similar to how vector spaces are defined as tuples (V,+,.) and then just talked about with V. RVs are talked about a lot as if they were really (X,P). Makes sense? $\endgroup$ – spelufo Apr 29 '14 at 15:54
  • $\begingroup$ Yes! The RV is both the mapping $X$ and the measure $P$! $\endgroup$ – PA6OTA Apr 29 '14 at 15:55

I have never gotten a clear answer on this question. My intuition is the same as yours. And it appears that it is the same in "hierarchical models" in statistics, where

$$ x \sim (X|Y)$$

is shorthand for:

$$ y \sim Y \\ x \sim (X | Y = y)$$

The latter shorthand would appear to indicate that there is a tripleable adjunction induced by "sampling", so that the conditioning operator $|$ is a monadic functor on random variables.

But I think that doesn't work out in the case of real random variables, because of measure theory.

  • $\begingroup$ I think in your example $X \sim (X|Y)$ is used in a different way. It specifies the distribution of X given the (already specified) distribution of $Y$. For example, if $Y$ is Uniform$[0,1]$ then we can define $X \sim Bernoulli(Y)$ and $X$ will have some new distribution. Et cetera. The discussion above concerns the case we already know the joint distribution of $(X,Y)$ $\endgroup$ – PA6OTA Apr 29 '14 at 16:04
  • $\begingroup$ Ah, I think I understand your point. Since, in the original discussion, we are given the distribution of $(X,Y)$. In this case, is there nothing at all we can say about $(X|Y)$ as shorthand for $y \sim Y; (X|Y=y)$? Surely, with the joint distribution we can marginalize on $X$ to recover the distribution of $Y$ (or am I mistaken there? -- /me gets Feller) $\endgroup$ – nomen Apr 29 '14 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.