# Joint Shannon entropy.

Let $X$ and $Y$ represent random variables with associated probability distributions $\{p(x_i)\}_{i=1}^n$ and $\{p(y_i)\}_{i=1}^n$, respectively; $X$ is the input source while $Y$ is the output effect.

Consider a communication channel, which is represented by a matrix of the transition probabilities (the so-called channel matrix): $$\mathcal C=\left( \begin{array}{cccc} p(y_1|x_1)& p(y_2|x_1) & \ldots & p(y_n|x_1) \\ p(y_1|x_2)& p(y_2|x_2) & \ldots & p(y_n|x_2) \\ \vdots & \vdots & \ddots &\vdots \\ p(y_1|x_n)& p(y_2|x_n) & \ldots & p(y_n|x_n) \\ \end{array} \right)$$ where $p(y_j|x_i)$ is the conditional probability of obtaining output $y_j$ given that the input is $x_i$. Each row of the channel matrix must sum to unity, namely: $$\sum_{j=1}^n p(y_j|x_i)=1 \quad \forall i \, .$$

The joint Shannon entropy (in bits) of two variables $X$ and $Y$ is defined as $$H(X,Y)=-\sum _{{i=1}}^n p(x_i) \,\sum _{j=1}^n p(y_j|x_i) \log _{2}[p(x_i)\, p(y_j|x_i)]\!\, .$$ Matrix $\mathcal C$ is a square matrix and might be diagonalizable. In this case, i.e. if it is diagonalizable, there exists an invertible matrix $\mathcal P$ such that $\mathcal P^{−1}\, \mathcal C\, \mathcal P$ is a diagonal matrix (whose entries are the eingenvalues of $\mathcal C$). Let $\mathcal D$ that diagonal matrix.

Since $\mathcal C$ and $\mathcal D$, are in general two different matrices, the calculation of $H(X,Y)$ will return different values using the $\mathcal C$ or $\mathcal D$ entries. Nevertheless, $\mathcal C$ and $\mathcal D$ are linked by diagonalization process and I wonder if the two entropies ($H$ for $\mathcal C$ or $\mathcal D$) are related is some way. I don't have the answer for this question, I tried to search but I didn't find any relation. Thanks in advance!

• I doubt very much that the diagonalization of the transition matrix plays nicely with entropy calculation. "$\mathcal C$ and $\mathcal D$ are in general two different matrices" : not only that: the diagonal matrix is not a transition matrix, so it would make little sense to compute a conditional entropy using it. I don't think this leads anywhere. – leonbloy May 12 '17 at 0:22