Say you have a set of random variables that have some mutual information structure. Could be that they all have nonzero MI between them. Or perhaps there are some clusters of variables with within-group high MI and between-group zero MI. In any of these cases, is it possible to map your dependent RV's to a (smaller) set of independent RV's?
My intuition is that mutual information between your RVs implies you have fewer degrees of freedom than number of RVs. So maybe you can map to a different space with new RVs corresponding to the true degrees of freedom. Perhaps there is a matrix reducing operation that does this.
The example I have in mind is an image, with each pixel being your original RVs. If you distort the image with a (perhaps non-uniform) convolution, you end up with dependence between your pixels (so that if you have many images of the same thing, but convolve each of them in the same manner, the post-processed pixels will have correlated values / mutual information). But you could replace the convolved image with fewer, larger pixels (say, just group them by high MI) and have the same approximate image but now with independent pixels.