It is known fact for random variables $(X,Y) \sim p(x,y)=p(x)p(y|x)$ the mutual information is concave function of $p(x)$ for fixed $p(y|x)$.

I have two confusions in interpreting the above fact:

1) when it says "....concave function of $p(x)$", does that mean function of the probability of one particular realization of $X$ i-e $x$ ?

2) If yes, let us suppose $p(x)$ is dependent on certain parameter $\theta$ i-e $p(x;\theta)$. When $\theta$ increases probability of one particular realization $x$ gets decreased i-e $p(X=x)$ decreases as $\theta$ increases. How the concavity of mutual information between $X$ and $Y$ relates to $\theta$?

  • 2
    $\begingroup$ No, mutual information is concave in the distribution on $X$, not a particular realisation of it. Consider discrete valued RVs. Note that the distributions in $X$ are $|\mathcal{X}|$-dimensional vectors in the appropriate simplex. Further, the mutual information is a function of the distributions $p_X, p_{Y|X}$. Now, if I hold $p_{Y|X}$ constant, there's some $f : \Delta^{|\mathcal{X}|} \to \mathbb{R}_+$ such that $I(X;Y) = f(p_X)$, where $p_X$ is a distribution on $\mathcal{X}$. The statement above says that $f $ is a concave function. $\endgroup$ – stochasticboy321 Jun 6 '16 at 2:20
  • $\begingroup$ I loved the perfect notations you used to describe the situation. (Though it took me a day to understand it.) So, is it correct to conclude that $I(X;Y)$ remains concave when $p_X$ is varied by its some inherent parameter while $p_{Y|X}$ remains fixed? $\endgroup$ – kaka Jun 7 '16 at 6:18
  • 1
    $\begingroup$ Sorry about that, it's just that for me, beating my head on an exact but dense statement usually helps more, especially when it comes to conceptual issues such as this. Hope this helped anyway. Re. your follow up, $I$ would remain concave in $p_X$ no matter how it's varied. However, if your question is whether $I(X;Y)$ is concave in some parameter $\theta$, that is not necessarily true, and requires at least convexity conditions on $p_X(\cdot;\theta)$. $\endgroup$ – stochasticboy321 Jun 8 '16 at 19:40

when it says "....concave function of $p(x)$", does that mean function of the probability of one particular realization of $X$?


A concave function has the property $f(\frac{a+b}{2}) \ge \frac{f(a)+f(b)}{2}$ [*]

In the case of the mutual information, the variables ($a,b$ above) are the probabilities densities themselves. So that would translate as

$$ I_{p_m(x)} \ge \frac{I_{p_1(x)}+I_{p_2(x)}}{2} $$

where $p_m(x) =\frac{p_1(x)+p_2(x)}{2}$. In words: if we construct a (weighted) average of two density functions, then the mutual information of the averaged density is greater than the (weighted) average of the original mutual informations.

[*] Replace for strict inequality if strictly concave. Further, notice that the inequality must also be true for other linear combinations, $\alpha a + (1-\alpha) b$, I'm using a particular case for the sake of illustration)

  • $\begingroup$ thanks. Your answer made me understand what stochasticboy321 had said in his comment. $\endgroup$ – kaka Jun 7 '16 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.