# Questions tagged [mutual-information]

Mutual Information is a metric used in Information Theory. It describes the amount of information shared by two random variables. Extensions for larger number of variables exist. See https://en.wikipedia.org/wiki/Mutual_information

43 questions
Filter by
Sorted by
Tagged with
51 views

• 55
15 views

• 55
62 views

### Prove that something that can be learned from $S=\sum_{i=1}^n x_i$ is not less than what can be learned from $S+x_{n+1}$

Is there a way to formally prove that whatever can be learned about any $x_i$ from $S=\sum_{i=1}^n x_i$ is certainly not less than what can be learned from $S'=S+x_{n+1}$ where $x_i$s belong to some ...
• 19
69 views

### Capacity of a 2-user discrete memoryless channel

I am trying to understand the capacity of the following 2 user multiple access channel with input and output alphabets $\mathcal{X}_{1}=\mathcal{X}_{2}=\mathcal{Y}=\{0,1\}$. The channel transition ...
• 55
1 vote
25 views

### Capacity of a discrete memoryless channel with infinite input alphabet size

I have been reading "Elements of Information Theory"- Cover & Thomas. It states the following: Definition: A discrete channel, denoted by $(\mathscr{X}, p(y \mid x), \mathscr{Y})$, ...
• 55
20 views

### Multivariate normal distribution; joint entropy

I have data that is the combination of two classes of multivariate normal distribution. I have used em algorithm to get mean, standard derivation, and weights for them. Could anyone let me know how to ...
37 views

Hi I have a really basic question about information theory : Reading Pattern Recognition and Machine Learning by Christopher Bishop he says ( in chapter 1.6) "If we are told that a highly ...
• 1,778
21 views

• 111
70 views

1 vote
44 views

### Can information transmission be proven in a Rule 30 ECA?

(This is hopefully a clearer version of an earlier post of mine.) I have been spending lots of time on the open challenge of proving the aperiodicity of the central column of a rule 30 cellular ...
• 4,979
145 views

### Mutual Information for Continuous Random variables for system with 1 input and multiple outputs (SIMO System)

Let's consider channels of the form $$\label{eq:channelmodel2} Y_{1}= X e^{j \theta_{1}}+ V_1$$ \label{eq:channelmodel3} Y_{2}=X e^{j \theta_{2}}+ V_2 \end{...
1 vote
65 views

### prove $I(X_1;X_2) \ge I(F_1(X_1);F_2(X_2))$

Suppose $X_1,X_2$ are random variables. Discrete. Also, assume $Y_1 =F_1(X_1), Y_2=F_2(X_2)$. Prove the following relation. $$I(X_1;X_2) \ge I(Y_1;Y_2)$$ I think the solution like this. Is it right? ...
• 435
46 views

### Geometric Interpretation of negativity of interaction information

The interaction information (II) in 3D is defined as a generalization of the 2D mutual information (MI).  \begin{eqnarray} I(X:Y:Z) &=& I(X:Y) - I(X:Y|Z) \\ &=& H(X) + H(Y)+H(Z) - H(...
• 2,112
### What is the simplest proof that the mutual information $I(X:Y)$ is always non-negative?
What is the simplest proof that mutual information is always non-negative? i.e., $I(X;Y)\ge0$