# Questions tagged [mutual-information]

Mutual Information is a metric used in Information Theory. It describes the amount of information shared by two random variables. Extensions for larger number of variables exist. See https://en.wikipedia.org/wiki/Mutual_information

65 questions
Filter by
Sorted by
Tagged with
14 views

### Mutual information of the joint distribution and the marginal distribution if two random variables are independent?

if $Y \rightarrow X$ and $Y \rightarrow Z$, is it possible to find the equality/inequality between $$I(X,Z; Y)$$ and $$I(X;Y) + I(Z;Y)$$ Or equivently, given $I(X,Z;Y)= I(X; Y) + I (Z; Y |X)$, ...
33 views

### Joint Probability Distribution between columns of Haar Random Unitary

Let U be a unitary matrix sampled from the Haar measure over U(D). We can decompose U as: \begin{equation} U = (U_{1}, \cdots , U_{D}) \end{equation} where the $i^{th}$ column is expressed as a ...
60 views

### Consider a fair coin flip. What is the mutual information between the top side and the bottom side of the coin?

Mutual information between the top side and the bottom side of the coin?. T is the top side, B is the bottom side. $$I(T;B) = H(B) - H(B|T) = \log(2) = 1$$ the log base is 2. I don't know why the ...
39 views

### Is Entropy of a Discrete Random Variable Condition on a Continuous Random Variable Always bigger than or Equal to Zero? [closed]

Let $X$ be a discrete random variable and $Y$ be a continuous random variable. Is the conditional entropy of $X$ given $Y$ always positive, i.e., $H(X|Y)\ge0$?
12 views

1 vote
66 views

### What is the mutual information between ingredients of a mixture of Gaussians?

Consider 2 random variables, $X$ and $Y$. $X$ is discrete and $Y$ is continuous. In particular, we have a Gaussian distribution $Y$ with mean $X$ and variance $\sigma = 1$, and $P(X=1) = \frac{1}{4}$...
1 vote
32 views

### Total Correlation is difference of relative entropies in general?

Motivation Consider a finite set $[q]=\{1,\dots,q\}$, random variables $X_1,\dots,X_k\in[q]$, and their product $X=X_1\otimes\cdots\otimes X_k\in[q]^k$, i.e. the components of $X$ are independent. Let ...
17 views

### Computing covariances of features/landmarks in environment in scan-matching algorithm

I am reading a paper titled "Real-Time Correlative Scan Matching" which explains a scan-matching algorithm used to obtain pose-pose constraints in pose-graph Simultaneous Localization And ...
34 views

205 views

### The tight bound for conditional mutual information: how much could conditional mutual information be greater than mutual information?

Given random variables $X$,$Y$ and another random variable $Z$, it is known that there are cases when the conditional mutual information $I(X;Y|Z)$ is greater than mutual information $I(X;Y)$. For ...
77 views

### Understanding an equality involving Conditional Mutual Information

It is stated in my information theory textbook that the conditional mutual information of (discrete) random variables $X, Y$, given $Z$ is defined as $I(X; Y|Z) = H(X|Z) - H(X| Y, Z)$, which is equal ...
1 vote
41 views

53 views

### What are the state-of-the-art dependence measures?

It is well known that mutual information (MI) is a widely-used measure for quantifying statistical dependence between two random variables. Also, I read about other measures such as distance ...
59 views

### Understanding problem of chain Rule for mutial information

So I have to proof $I(X;Z|Y) = I(Z;Y|X) - I(Z;Y) + I(X;Z)$ Write mutual information in terms of entropy or use the chain rule for mutual information for an immediate proof. So I want to directly ...
54 views

### Stochastic Mutual Information Estimator

I am reading https://openreview.net/forum?id=ByxaUgrFvH and do not understand why they need a "complicated" derivation, because it seems to follow immediately. Problem Let $\mathbf{x}$ be a ...
1 vote
42 views

### Proof of the information bottleneck equations

In The Information Bottleneck Method, the third term of Eq.(31) is $P_{t+1}(y|\tilde{x})=\sum_yp(y|x)p_t(x|\tilde{x})$, which minimizes the term $D_{KL}[p(y|x)|p(y|\tilde{x})]_{<p(x,\tilde{x})>}$...
80 views

### Mutual Information of Vectors with Large Inner Product

If we have a joint distribution of two (complex) vectors $x,y\in \mathbb{C}^d$ of norm $1$ such that their inner product $\langle x|y\rangle$ is $1-\epsilon$, can we lower bound the mutual information ...
96 views

### Why does $H(X|Y)$ equal the "missing information" of $Y$ about $X$?

I've seen mentioned in (Horodecki, Oppenheim, Winter 2005) the fact that the conditional information equals the amount of information that Alice needs to send Bob in order for him to fully reconstruct ...
75 views

### Encoding $I(X;Y)$ into a random variable $Z$ such that $H(Z) = I(X;Y)$ and $I(X;Z) = I(Y;Z) = I(X,Y)$

Is it possible to encode the mutual information $I(X;Y)$ between two random variables $X$ and $Y$ into another random variable $Z$, such that $Z$ "contains" exactly the information that $X$ ...
44 views

### Mutual Information and Entropy calculation

It is well known that Shannon's joint entropy ($H(X,Y)$) as well as mutual information ($I(X;Y)$) between two variables $X$ and $Y$ are non-negative based on Jensen's inequality. I read in a source ...
27 views

Consider a random variable $X$ taking values in $\{0,1,\ldots,n\}$ and $Y$ takes values in $\{0,1\}$. Let $a_{i}=P\left(X={i}\right), b_{j}=P(Y=j)$. Also, $p_{i}=P\left(Y=0 \mid X=x_{i}\right), q_{i}=... 0 votes 0 answers 72 views ### Maximizing the variance of weights of Bernoulli RV maximize mutual information? I have a random variable$X=a_1X_1+a_2X_2 + \ldots a_kX_k$where$X_i \sim Bern(q)$,$X_i \perp X_j, \forall i,j\in \{1,2\ldots,k\}$. Also$\sum_{i=1}^{k} a_i=k$and$a_i \in \mathbb{N} \bigcup \{0\...
1 vote
A random unitary channel (RUC) acting on a quantum system $S$ is any completely positive and trace preserving (CPTP) linear map $\mathcal{E}$ that can be expressed as \begin{align} \mathcal{E}(\...