0
$\begingroup$

It is well known that mutual information (MI) is a widely-used measure for quantifying statistical dependence between two random variables. Also, I read about other measures such as distance correlation, mutual information dimension, Hilbert-Schmidt independence criterion (HSIC), Kernel-based Constrained Covariance (COCO), etc.

My question is: what are the current state-of-the-art dependence measures other than the ones I mentioned?

$\endgroup$

1 Answer 1

0
$\begingroup$

I believe there is no such thing as state-of-the-art here, it might completely depend on the problem you are looking at! For example, are you working with images, are you solving an inference problem with a particular set of assumptions, ...? Those question might help guiding towards which measure to use.

What you will often see is that people in the information theory community use Mutual Information (but you can think of it as the KL divergence really since $I(X;Y)=D_{KL}(P_{X, Y}\|P_XP_Y)$), because it is an object that is pretty well understood, and we know how to compute it in many cases.

But as you mentioned, there are so many other possibilities! One measure of independence I find particularly interesting stems from the family of divergences called $f$-Divergences. From these, you can define a measure of independence by looking at $D_f(P_{X, Y}\|P_XP_Y)$, in the same way as the mutual information is simply $D_{KL}(P_{X, Y}\|P_XP_Y)$. The powerful thing is that here, for each choice of convex $f$, $D_{KL}(P_{X, Y}\|P_XP_Y)$ will give you some measure of independence in the end. With this family, you can retrieve many well-known divergences like KL divergence, Total Variation Distance, Hellinger Divergence, ...

I would say as a rule of thumb, look first at the problem you are trying to solve, and then figure out which measure(s) of independence would make sense for that particular case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.