# Derivative of mutual information

Here is the definition of mutual information

$$I(X;Y) = \int_Y \int_X p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } \, \mathrm d x \, \mathrm d y$$

where $X$ and $Y$ are two random variables, $p(x)$ and $p(y)$ are their PDFs, and $p(x,y)$ is the joint PDF.

I am wondering what is the derivative of $I(x;y)$ with respect to $X$, or $Y$? Namely,

$$\frac{\mathrm d}{\mathrm dX} I(X;Y) = \, ?$$ $$\frac{\mathrm d}{\mathrm dY} I(X;Y) = \, ?$$

Thanks.

• How is this "Derivative" defined? X and Y are random variables, not scalars. – Batman Sep 25 '14 at 13:53
• Random raviables have PDF. It is a function. So it is possible it is differentiable – Ono Sep 25 '14 at 14:57
• But I(X;Y) is not a function of the dummy variables $x,y$ on the right hand side -- $I(X;Y)$ is a number. – Batman Sep 25 '14 at 15:01
• It is a number, but not a fixed one. It changes as x or y changes. – Ono Sep 25 '14 at 15:02
• No, it does not. – Danny W. Sep 25 '14 at 17:00

The notation $I(X;Y)$ does not mean that the information is a function of the variable $X$. In that sense, it's just a number, hence its derivative is zero. If you want an analogy, think of the expectation of a random variable $E(X)$ : it does not "depend on" $X$, it's a number; it would not make sense to ask about $d E(X)/dX$
The mutual information $I(X;Y)$ is actually a functional of the random variables $X$ and $Y$, therefore it might have a functional derivative or a Fréchet derivative.