What does it mean to differentiate entropy and information contents?

Question

I'm taking an information theory class and encountered some concepts regarding differentiation that I wasn't sure how to understand.

In class, my instructor gave many examples and solutions done by "finding the derivative" for certain probability distribution functions, or finding the derivatives for other concepts like information entropy or mutual information.

But in general, what does it exactly mean to differentiate a probability, and what is the motivation for doing so?

$\space$

Edit

Here's a specific example to what I'm referring to. It's an exercise problem from the textbook Information Theory, Inference, and Learning Algorithms - David Mackay.

$\space$

Exercise Problem

Consider the $Z$ channel with $f = 0.15$. Identifying the optimal input distribution is not so straightforward. We evaluate $I(X;Y)$ explicitly for $P_X = \{p_0, p_1\}$. First, we need to compute $P(y)$. The probability of $y = 1$ is easiest to write down:

$$P(y = 1) = p_1(1 - f)$$

Then the mutual information is:

$$I(X; Y) = H(Y) - H(Y|X)$$ $$ = H_2(p_1(1 - f)) - (p_0H_2(0) + p_1H_2(f))$$ $$ = H_2(p_1(1 - f)) - p_1H_2(f))$$

This is a non-trivial function of $p_1$. It is maximized for $f = 0.15$ by $p_1^* = 0.445$. We find capacity $C(Q_Z) = 0.685$.

Now, in the case of general $f$, show that the optimal input distribution is

$$p_1^* = \frac{1/(1-f)}{1 + 2^{(H_2(f)/(1 - f))}}$$

$\space$

Solution

In the exercise, we showed that the mutual information between input and output of the $Z$ channel is

$$I(X;Y) = H(Y) - H(Y|X)$$ $$ = H_2(p_1(1 - f)) - p_1H_2(f)$$

We differentiate this expression with respect to $p_1$, taking care not to confuse $\log_2$ with $\log_e$:

$$\frac{d}{dp_1}I(X;Y) = (1 - f)\log_2\frac{1 - p_1(1 - f)}{p_1(1 - f)} - H_2(f)$$

Setting this derivative to zero and rearranging it, we obtain:

$$p_1^*(1 - f) = \frac{1}{1 + 2^{H_2(f)/(1 - f)}}$$

so the optimal input distribution is

$$p_1^* = \frac{1/(1-f)}{1 + 2^{(H_2(f)/(1 - f))}}$$

$\space$

So that's the particular question and solution that motivated me to ask this question. I just wasn't sure how the solution came to the idea of differentiating the mutual information, and if there was a more "general" motivation for differentiating probability functions what they might be.

Thank you.

Answer 1

The author is not differentiating "information".

Channel capacity, mutual information, etc. are probability dependent functions. He is differentiating to find the maximum of the mutual information which is by definition the channel capacity with the maximization performed over all input distributions.

In this case the mutual information happens to depend only on one parameter $f,$ so the maximization can be performed analytically.

In the general case, the Blahut-Arimoto algorithm is used for this maximization.