2
$\begingroup$

$I(X;Y)$ is the symbol for mutual information, where $X$ and $Y$ are random variables. It measures the co-dependence between $X$ and $Y$ including non-linear interaction, whereas linear correlation is written as $\rho(X,Y)$.

Why does mutual information have a semicolon between its inputs, whereas the arguments for correlation only have a comma? How incorrect is it to write $I(X,Y)$ instead?

$\endgroup$

2 Answers 2

3
$\begingroup$

You asked two questions, let me answer them both

Why do we use semicolons for mutual information?

While with two variables it would be fine to write a comma (people would understand what you mean), you would get into an issue pretty soon when you are dealing with vectors, or mutual information between sets of random variables. Most notably, in many inequalities one uses in information theory (say Fano's, or channel capacity calculations), you will encounter mutual information terms like

$I(X; Y,Z)$ and $I(X_1, X_2; Y_1, Y_2)$

and you "can" of course write them as

$I(X, (Y,Z))$, and $I((X_1, X_2), (Y_1, Y_2))$

but it is that much uglier.

Personally, I like the semicolon since it emphasizes the two halves of mutual information; it is not a function that takes more than two inputs (usually; there is a multi-way information, but that is much less used), and in English, when we list more than two objects, we tend to use commas and not semicolons as separators (unless you are listing sentences, but you should probably just divide that up then). The main issue I have with it is that in statistics, semicolons sometimes are used to denote parameters (more so in frequentist setting; in Bayesian, you usually use a $\mid$ for a conditional observation), but overall, I like this notation.

How incorrect is it?

It is incorrect. Don't do it. If you really want to use this notation, define in the beginning of your document I(X, Y) := I(X; Y), but please don't do it.

$\endgroup$
7
  • $\begingroup$ how come correlation gets away with the comma $\endgroup$
    – develarist
    Sep 9, 2020 at 21:57
  • $\begingroup$ @develarist they tend to not compute correlation between sets of random variables; it is usually just between two random variables $\endgroup$
    – E-A
    Sep 9, 2020 at 21:59
  • $\begingroup$ is there a name by definition for the $;$ operator vs the $,$ operator? otherwise, do i just refer to both as parsers? $\endgroup$
    – develarist
    Sep 9, 2020 at 22:01
  • $\begingroup$ I would probably not call them operators (though in some abstract sense, I guess you can argue everything is an operator like what I assume category theorists do in their free time); I like the word separator, but parser should work as well. I am not a linguist though. $\endgroup$
    – E-A
    Sep 9, 2020 at 22:04
  • 2
    $\begingroup$ No, multi-way information I mention would be like $I(X_1; X_2; X_3; X_4)$, and it can be negative and it is a completely different beast than mutual information. You can see the wiki on it here: en.wikipedia.org/wiki/Multivariate_mutual_information (Total correlation is a much nicer quantity being non-negative an interpretable, and one can write that as a difference of various entropy terms.) $\endgroup$
    – E-A
    Sep 9, 2020 at 22:10
0
$\begingroup$

$I(X;Y) = I(Y;X)$, so technically you should be fine in writing $I(X,Y)$. However, based on what I gather from wiki (https://en.wikipedia.org/wiki/Mutual_information), $I(X;Y)$ could be used to emphasize the second interpretation in terms of KL divergence which highlights the asymmetry introduced by conditioning over one of the two variables (Ref: second expression in the Relation to Kullback–Leibler divergence section)

$\endgroup$
2
  • $\begingroup$ in the context of functions that accept two random variables for inputs, what is the meaning of using ; instead of , ? $\endgroup$
    – develarist
    Sep 9, 2020 at 21:12
  • $\begingroup$ As I understand, comma is generally used to imply a logical AND. So in that sense, comma is not the right operator for mutual information since although reciprocal the relationship between X and Y is not of AND. So in a general case, we could very well have $I(X_1,X_2,X_3...X_n; Y_1, Y_2,...Y_m)$ which would be the entropy reduction due to knowing the joint distribution of $X_1,X_2,X_3...X_n,Y_1, Y_2,...Y_m$ as opposed to knowing the two joint distributions $X_1,X_2,X_3...X_n$ and $Y_1, Y_2,...Y_m$ separately $\endgroup$
    – Kartik
    Sep 9, 2020 at 23:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .