I can calculate the mutual information between two, (possibly multivariate) Gaussian random variables $I[X,Y]$. Is there anything I can say about the mutual information (perhaps establish a bound) between $I[X, f(Y)]$, where $f$ is some nonlinearity? I understand the the data processing inequality establishes $I[X,Y]$ as an upper bound, but is there a stronger statement I can make given that $X$ and $Y$ are Gaussian and if the nonlinearity is sufficiently "simple" (for example, a squared nonlinearity)?


If $f$ is a 1-1 function then $f(Y)$ gives you the same info as $Y$ and so $I(X;Y)=I(X; f(Y))$. Formally, since you can invert $f$, we have by multiple uses of the data processing inequality $$ I(X;Y)\geq I(X;f(Y)) \geq I(X;f^{-1}(f(Y))) = I(X;Y)$$

  • $\begingroup$ How about if $f$ is no longer one-to-one (such as a thresholded function, or a squared nonlinearity)? $\endgroup$ – user2452976 Nov 9 '17 at 19:58
  • $\begingroup$ Another case is when $X,Y$ are independent, then $X, f(Y)$ are also independent (regardless of the function $f$, even if $f$ is not 1-1) so $I(X;Y)=I(X;f(Y))=0$. $\endgroup$ – Michael Nov 9 '17 at 20:00
  • $\begingroup$ It is hard to make general statements for more interesting cases, likely you should just try computing your own particular examples. $\endgroup$ – Michael Nov 9 '17 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.