I'm taking an information theory class and encountered some concepts regarding differentiation that I wasn't sure how to understand.

In class, my instructor gave many examples and solutions done by "finding the derivative" for certain probability distribution functions, or finding the derivatives for other concepts like information entropy or mutual information.

But in general, what does it exactly mean to differentiate a probability, and what is the motivation for doing so?



Here's a specific example to what I'm referring to. It's an exercise problem from the textbook Information Theory, Inference, and Learning Algorithms - David Mackay.


Exercise Problem

Consider the $Z$ channel with $f = 0.15$. Identifying the optimal input distribution is not so straightforward. We evaluate $I(X;Y)$ explicitly for $P_X = \{p_0, p_1\}$. First, we need to compute $P(y)$. The probability of $y = 1$ is easiest to write down:

$$P(y = 1) = p_1(1 - f)$$

Then the mutual information is:

$$I(X; Y) = H(Y) - H(Y|X)$$ $$ = H_2(p_1(1 - f)) - (p_0H_2(0) + p_1H_2(f))$$ $$ = H_2(p_1(1 - f)) - p_1H_2(f))$$

This is a non-trivial function of $p_1$. It is maximized for $f = 0.15$ by $p_1^* = 0.445$. We find capacity $C(Q_Z) = 0.685$.

Now, in the case of general $f$, show that the optimal input distribution is

$$p_1^* = \frac{1/(1-f)}{1 + 2^{(H_2(f)/(1 - f))}}$$



In the exercise, we showed that the mutual information between input and output of the $Z$ channel is

$$I(X;Y) = H(Y) - H(Y|X)$$ $$ = H_2(p_1(1 - f)) - p_1H_2(f)$$

We differentiate this expression with respect to $p_1$, taking care not to confuse $\log_2$ with $\log_e$:

$$\frac{d}{dp_1}I(X;Y) = (1 - f)\log_2\frac{1 - p_1(1 - f)}{p_1(1 - f)} - H_2(f)$$

Setting this derivative to zero and rearranging it, we obtain:

$$p_1^*(1 - f) = \frac{1}{1 + 2^{H_2(f)/(1 - f)}}$$

so the optimal input distribution is

$$p_1^* = \frac{1/(1-f)}{1 + 2^{(H_2(f)/(1 - f))}}$$


So that's the particular question and solution that motivated me to ask this question. I just wasn't sure how the solution came to the idea of differentiating the mutual information, and if there was a more "general" motivation for differentiating probability functions what they might be.

Thank you.

  • $\begingroup$ That's hard to answer "in general". "my instructor gave many examples and solutions... finding the derivatives for other concepts ..." That's too vague. Some concrete examples? $\endgroup$ – leonbloy Jun 17 '18 at 14:00
  • $\begingroup$ Sorry. I'll edit it into the original question. $\endgroup$ – Seankala Jun 17 '18 at 14:02
  • $\begingroup$ This has little to do with entropy or information. You are supposed to know (from Calculus) that the standard way to find extrema (local, hopefully global) of any differentiable function is to find where the derivative is zero. $\endgroup$ – leonbloy Jun 18 '18 at 17:55

The author is not differentiating "information".

Channel capacity, mutual information, etc. are probability dependent functions. He is differentiating to find the maximum of the mutual information which is by definition the channel capacity with the maximization performed over all input distributions.

In this case the mutual information happens to depend only on one parameter $f,$ so the maximization can be performed analytically.

In the general case, the Blahut-Arimoto algorithm is used for this maximization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.