Let's assume I have three probability distributions A,B,C.

The entropy of each is 1.58, with joint entropy 1.58.

Calculating mutual entropy with formula I(A,B,C) = H(A)+H(B)+H(C)-J(A,B,C) results in 3.16.

How could this be explained? This means there is more common information than there is a capacity of a channel. Could you explain this?


The mutual information $I(X;Y)$ (not "mutual entropy"; also, notice the semicolons, don't confuse them with commas) relates always two (perhaps multivariate) random variables.

$I(A;B;C)$ is a highly dubious extension of the concept of the "real" mutual information, which is not very meaningful, nor much used. For one thing, as you noted, it can be negative. See also here.

  • $\begingroup$ Thank you @leonbloy $\endgroup$ – filtertips Feb 8 '19 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.