# Questions tagged [mutual-information]

Mutual Information is a metric used in Information Theory. It describes the amount of information shared by two random variables. Extensions for larger number of variables exist. See https://en.wikipedia.org/wiki/Mutual_information

83 questions
Filter by
Sorted by
Tagged with
51 views

### Mutual information expansion not justifiable

I have recently read a mutual information term, $I(X;Y,Z)=E_{p(X,Y,Z)}\big[\log\frac{ p(X|Y,Z)}{p(X)}\big]$. While this expansion does not make sense to me. Is it correct? My understanding (using ...
• 164
31 views

### Mutual information between randomly permuted variables

I am trying to understand how to calculate the mutual information between two variables which have nonzero conditional mutual information (when conditioned on a third variable) but (I believe) zero ...
• 406
29 views

### How to find "extra" mutual information

Codewords $00$ and $11$ are sent with equal probability through a BSC with error probability p. Compute the mutual information between the codeword sent and the first digit received as output. I have ...
• 116
71 views

### Mutual information when sampling a random variable multiple times

Let $X$ be a random variable. For a fixed (known) preparation of $X$, suppose I have a protocol that generates a second random variable, $Y$, in a way that indirectly depends on $X$. Ultimately the ...
• 699
36 views

### Estimating the conditional entropy of a discrete variable conditioning on continuous variable

I am doing a machine learning project and I am trying to select the best features by computing their mutual information and select the ones with the highest information gain. I was looking at this ...
• 1,717
23 views

### Maximum Capacity of a Communication Channel w.r.t. $P(Y | X)$ when $X$ and $Y$ are discrete

Let $X$ and $Y$ be two discrete stochastic variables. I want to find $P(Y|X)$ that maximizes the mutual information between $X$ and $Y$, i.e., $$\max_{P(Y|X)} I(X, Y).$$ This problem is different from ...
• 357
32 views

• 738
36 views

### interpretation of negative pointwise mutual information

Since: $$pmi(x,y)\quad =\quad \log(\frac{P(X=x,Y=y)}{P(X=x)P(Y=y)})\quad =\quad -t$$ iff $$2^{t}P(Y=y | X=x) \quad =\quad P(Y=y)$$ the interpretation of a negative PMI seems very clear to me. So I ...
• 73
35 views

### Maximizing the expectation in CE Importance sampling.

Suppose the following maximization: $$v_t = \arg \max_{v} E_{v_{t-1}} 1\{S(x) \geq \gamma\} \frac{f(x;u)}{f(x;v_{t-1})}\ln f(x;v) = max_{v} E_{v_{t-1}} 1\{S(x) \geq \gamma\} W(u;v_{t-1}) \ln f(x;v),$$ ...
• 147
1 vote