I'm reading about independent component analysis (ICA) where the goal is to find an unmixing matrix to maximize independence in latent, linearly mixed sources.

The paper I'm reading (Hyvarinen) says that mutual information is a natural benchmark for this because it is the information-theoretic measure of independence of random variables. It goes on to say that mutual information can be expressed using the concept of negentropy so that maximizing negentropy is roughly equivalent to minimizing mutual information. This is where I'm getting lost.

Terms in paper:

$$\textbf{differential Entropy: }H(\textbf y) = - \int f(\textbf y) \log f(\textbf y) d \textbf y $$ $$\text{with random vector } \textbf y=(y_1,...,y_n)^T \text{ and density } f(.) $$ $$\textbf{negentropy: } J(\textbf y ) = H(\textbf y_{gauss} ) - H(\textbf y )$$ $$\text{Property: negentropy is invariant for linear transforms}$$ $$\text{mutual information can be expressed in terms of negentropy:}$$ $$\textbf{mutual information: } I(\textbf y) = J(\textbf y) - \sum_i J(y_i) \ \ \text{ (5)}$$

It goes on to say that,

"Because negentropy is invariant for invertible linear transformations, it is now obvious from (5) that finding an invertible transformation $\bf W$ that minimizes the mutual information is roughly equivalent to finding directions in which the negentropy is maximized."

I don't know if this is because I have not studied information theory at all, but this is not obvious to me. Would someone please attempt to give me an intuitive explanation for why this property of negentropy results in the above statement?

  • 1
    $\begingroup$ Any chance you have a link to the paper? $\endgroup$ – gary Dec 7 '17 at 19:03
  • $\begingroup$ Sure, the quote I'm referring to is on page 3: cs.helsinki.fi/u/ahyvarin/papers/TNN99new.pdf $\endgroup$ – Austin Dec 7 '17 at 19:04
  • 1
    $\begingroup$ I think I'm starting to understand through the illustration and explanation on this page: fourier.eng.hmc.edu/e161/lectures/ica/node5.html $\endgroup$ – Austin Dec 7 '17 at 19:29
  • 1
    $\begingroup$ I thought I was the only one here into this stuff. I am trying to understand Polychorics myself. $\endgroup$ – gary Dec 7 '17 at 19:40
  • 1
    $\begingroup$ Good luck, I'll see if I can understand the links so I can suggest something useful. good luck with it :). $\endgroup$ – gary Dec 7 '17 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.