2
$\begingroup$

Here is the definition of mutual information

$$I(X;Y) = \int_Y \int_X p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } \, \mathrm d x \, \mathrm d y$$

where $X$ and $Y$ are two random variables, $p(x)$ and $p(y)$ are their PDFs, and $p(x,y)$ is the joint PDF.

I am wondering what is the derivative of $I(x;y)$ with respect to $X$, or $Y$? Namely,

$$\frac{\mathrm d}{\mathrm dX} I(X;Y) = \, ?$$ $$\frac{\mathrm d}{\mathrm dY} I(X;Y) = \, ?$$

Thanks.

$\endgroup$
  • 2
    $\begingroup$ How is this "Derivative" defined? X and Y are random variables, not scalars. $\endgroup$ – Batman Sep 25 '14 at 13:53
  • $\begingroup$ Random raviables have PDF. It is a function. So it is possible it is differentiable $\endgroup$ – Ono Sep 25 '14 at 14:57
  • 1
    $\begingroup$ But I(X;Y) is not a function of the dummy variables $x,y$ on the right hand side -- $I(X;Y)$ is a number. $\endgroup$ – Batman Sep 25 '14 at 15:01
  • $\begingroup$ It is a number, but not a fixed one. It changes as x or y changes. $\endgroup$ – Ono Sep 25 '14 at 15:02
  • 2
    $\begingroup$ No, it does not. $\endgroup$ – Danny W. Sep 25 '14 at 17:00
3
$\begingroup$

The notation $I(X;Y)$ does not mean that the information is a function of the variable $X$. In that sense, it's just a number, hence its derivative is zero. If you want an analogy, think of the expectation of a random variable $E(X)$ : it does not "depend on" $X$, it's a number; it would not make sense to ask about $d E(X)/dX$

In other sense, you could say that it's a function (or a functional) of the probability densities - but this is not what you are after.

$\endgroup$
2
$\begingroup$

The mutual information $I(X;Y)$ is actually a functional of the random variables $X$ and $Y$, therefore it might have a functional derivative or a Fréchet derivative.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.