# Comparison between Shannon's and Blackwell's measure of informativeness

I want to compare the concept of precision of information'' between signals $x \in X$ and states $\omega \in \Omega$ defined by Blackwell and Shannon.

Denote the conditional probability distribution over signals given states by $\mathcal{P}_{X|\Omega}$ and the unconditional probability distribution over states and signals by $\mathcal{P}_\Omega$ and $\mathcal{P}_X$ respectively. Let the number of states and signals be finite. Blackwell says that the conditional probability distribution $\mathcal{P}_{X|\Omega}$ is more informative than another distribution $\mathcal{\tilde{P}}_{X|\Omega}$, $\mathcal{P}_{X|\Omega} \supset \mathcal{\tilde{P}}_{X|\Omega}$, if and only if there exist a Markov matrix $M$: $$\mathcal{P}_{X|\Omega} M = \mathcal{\tilde{P}}_{X|\Omega}$$

For example, suppose the conditional probability distribution $\mathcal{P}_{X|\Omega}$ is given by the following Markov matrix: $$\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$$ Such a matrix is fully informative: The signal reveals that state with perfect accuracy. Suppose the matrix $M$ is written as: $$M = \begin{bmatrix} 1/2 & 1/2 \\ 1/2 & 1/2 \end{bmatrix}$$ The distribution $\mathcal{\tilde{P}}_{X|\Omega}$ is then very diffuse in the sense that the signals do not favour any state.

Shannon uses a different notion of informativeness''. Let entropy be given by function $H$. For the unconditional distribution over states, the entropy is:
$$H(\mathcal{P}_\Omega) = - \sum_\Omega \mathcal{P}_\Omega(\omega) \log_2 (\mathcal{P}_\Omega(\omega))$$ Informativeness between signals and states is quantified by the mutual information of the unconditional probability distribution between signals and states: $$I(\mathcal{P}_{X,\Omega}) = H(\mathcal{P}_\Omega) - \sum_X \mathcal{P}_X (x) H(\mathcal{P}_{\Omega|X} ( \ |x))$$ In Shannon's notion of informativeness, $\mathcal{P}_{X,\Omega}$ is more informative than $\mathcal{\tilde{P}}_{X,\Omega}$ if $I(\mathcal{P}_{X,\Omega}) > I(\mathcal{\tilde{P}}_{X,\Omega})$.

What is the relationship between the two concepts of informativeness is? Can one proof that an information structure which is more informative in the sense of one author is more informative in the sense of another author?

• so Blackwell makes no difference for a linear (with full rank) transformation of the signal ? is that what means there exists a matrix $M$ such that $\mathcal{P}_{X|\Omega} M = \tilde{\mathcal{P}}_{X|\Omega}$ ? – reuns Jan 15 '16 at 12:22
• @user1952009 : Can you please expand your question a bit? I don't understand it the way it is currently stated. I have changed the description of the relationship between the conditional probability distributions above a bit . – fabian Jan 15 '16 at 12:26
• can you give a concrete example of that relation $\mathcal{P}_{X|\Omega} M = \tilde{\mathcal{P}}_{X|\Omega}$ (defining the sets $X$ and $\Omega$ ) ? – reuns Jan 15 '16 at 12:32
• @user1952009 : Thank you for your comment. I added an example. Does that help? – fabian Jan 15 '16 at 14:51

Let $P_X$ and $\tilde{P}_X$ be column stochastic matrices (experiments) of dimension $n_i \times |\Omega|$, $i=1,2$. If $\exists$ a column stochastic matrix $M_{n_1\times n_2}$ s.t. $P_X=M\tilde{P}_X$ then $\tilde{P}_X$ is said to be Blackwell more informative than $P_X$. Denote by $\geq_B$ this partial ordering on left stochastic matrices (though it is more common to work with right stochastic matrices, I'm keeping the setting as in the question).
It can be shown that $\tilde{P}_X \geq_B P_X \Rightarrow I(\tilde{P}_X) \geq I(P_X)$, i.e. Blackwell more informative implies higher Shannon Mutual Information, though the converse is not true.
The implication is straightforward (but tedious) by working through the algebra: take a $\tilde{P}_X$ and an $M$, get the expression for each $P_{X,ij}$ and replace in the definitions, working through the inequalities.