Questions tagged [mutual-information]
Mutual Information is a metric used in Information Theory. It describes the amount of information shared by two random variables. Extensions for larger number of variables exist. See https://en.wikipedia.org/wiki/Mutual_information
65
questions
0
votes
0
answers
14
views
Mutual information of the joint distribution and the marginal distribution if two random variables are independent?
if $Y \rightarrow X$ and $Y \rightarrow Z$, is it possible to find the equality/inequality between
$$
I(X,Z; Y)
$$
and
$$
I(X;Y) + I(Z;Y)
$$
Or equivently, given $I(X,Z;Y)= I(X; Y) + I (Z; Y |X)$, ...
0
votes
0
answers
33
views
Joint Probability Distribution between columns of Haar Random Unitary
Let U be a unitary matrix sampled from the Haar measure over U(D). We can decompose U as:
\begin{equation}
U = (U_{1}, \cdots , U_{D})
\end{equation}
where the $i^{th}$ column is expressed as a ...
- 41
0
votes
1
answer
60
views
Consider a fair coin flip. What is the mutual information between the top side and the bottom side of the coin?
Mutual information between the top side and the bottom side of the coin?.
T is the top side, B is the bottom side.
$$I(T;B) = H(B) - H(B|T) = \log(2) = 1$$
the log base is 2.
I don't know why the ...
- 3
0
votes
1
answer
39
views
Is Entropy of a Discrete Random Variable Condition on a Continuous Random Variable Always bigger than or Equal to Zero? [closed]
Let $X$ be a discrete random variable and $Y$ be a continuous random variable. Is the conditional entropy of $X$ given $Y$ always positive, i.e., $H(X|Y)\ge0$?
- 81
0
votes
0
answers
12
views
Finding mutual information in discrete linear partial observation stochastic process
I have one basic question maybe is not to hard for you but I am a bit confused. Let our system be like this:
\begin{align}
X_{k+1} &= A_k X_k + W_k \\
Y_k &= C_k X_k + V_k
\end{align}
where $...
- 1
0
votes
0
answers
61
views
$f$-mutual information and chain rule
The standard definition of mutual information for a pair of random variables $X, Y$ is $I(X; Y) = D_{KL}(P_{X, Y} \| P_X \times P_Y)$. One of the most important properties of this definition is the ...
- 301
1
vote
0
answers
26
views
Mutual information of dependent variables
I am looking at a setting where I have a random variable $X$ and two other random variables that are derived from $X$, $Y = f(X)$ and $Z =g(X)$. What I want to do is compute the mutual information ...
- 57
0
votes
0
answers
20
views
Characterization of functions that have the data processing inequality
I wonder if people have studied the characterization of data processing inequality. Here's the data processing inequality for mutual information: 3 variables for Markov chain: $X\to Y\to Z$, then $I(X,...
1
vote
0
answers
66
views
What is the mutual information between ingredients of a mixture of Gaussians?
Consider 2 random variables, $X$ and $Y$. $X$ is discrete and $Y$ is continuous.
In particular, we have a Gaussian distribution $Y$ with mean $X$ and variance $\sigma = 1$, and
$P(X=1) = \frac{1}{4}$...
- 1,032
1
vote
1
answer
32
views
Total Correlation is difference of relative entropies in general?
Motivation
Consider a finite set $[q]=\{1,\dots,q\}$, random variables $X_1,\dots,X_k\in[q]$, and their product $X=X_1\otimes\cdots\otimes X_k\in[q]^k$, i.e. the components of $X$ are independent.
Let ...
- 3,174
0
votes
0
answers
17
views
Computing covariances of features/landmarks in environment in scan-matching algorithm
I am reading a paper titled "Real-Time Correlative Scan Matching" which explains a scan-matching algorithm used to obtain pose-pose constraints in pose-graph Simultaneous Localization And ...
0
votes
0
answers
34
views
Need help finding the stationary points of a function
Setup: I have a function
$$
f(x)
=-\delta\left(1-\frac{\sigma^2}{(x/\delta)+\sigma^2}\right)
+\delta\log\left(1+\frac{x}{\delta\sigma^2}\right)
+2I\left(X;\sqrt{\frac{1}{(x/\delta)+\sigma^2}}\cdot X+G\...
- 664
0
votes
1
answer
38
views
Intuition on Mutual information in XAI
Following wikipedia, mutual information $MI(X,Y)$ is 'how much knowledge is in the output of one random variable about the other variable'.
If MI between 2 variables is high, this does not mean I know ...
- 28
0
votes
0
answers
46
views
Derivative of a mutual information for a Gaussian channel
I saw this question which inspired me to ask mine. Given that I have a mutual information
$$
I\left(X;\sqrt{\frac{1}{(x/\delta)+\sigma^2}}\cdot X + Y\right),
$$
which is a function of $x$ that is not ...
- 664
0
votes
0
answers
20
views
$X \to Y \to Z$ PGM with $X,Y,Z ~ \text{MVN}(0,\Sigma)$. What is mutual information $I(X;Z)$? (Cover & Thomas 8.9)
Cover and Thomas Elements of Information Theory Problem 8.9 describes a 3-node network $X \rightarrow Y \rightarrow Z$ with a multivariate Gaussian distribution. C&T asks, what is the mutual ...
- 147
0
votes
0
answers
13
views
When should you use Mutual Information over Pearson Correlation?
I have a dataset with ~1000 features. I know that some of those features are leaking information from my target variable.
I'd like to find those features by checking similarity between all features ...
- 541
2
votes
2
answers
46
views
Why does the common information $C\equiv X\wedge Y$ give $H(X)=H(CX)$?
Given two random variables $X,Y$, their common information $X\wedge Y$ is defined in (Wolf and Wultschleger 2004) as the random variable $X\wedge Y=f_X(X)=f_Y(Y)$ constructed as follows:
Let $G$ be ...
- 6,483
0
votes
0
answers
19
views
the mutual information between the column vectors of a Haar unitary
Given a Haar random unitary $U=(U_1, \cdots, U_d) \in \mathbb{C}^{d\times d}$ where $U_i$ is the $i$-th column vector of $U$, what about the mutual information between $U_i$ and $U_j$ for any $i \ne j$...
- 141
8
votes
1
answer
887
views
Is there a chain rule for Sibson's mutual information?
Mutual Information satisfies the chain rule: $$I(X,Y;Z) = I(X;Z) + I(Y;Z|X).$$
The chain rule is useful and the proof is simply linearity of expectations.
Sometimes we want something stronger than ...
- 845
0
votes
0
answers
14
views
Derivation of the analytical solution for conditional mutual information
I am aware of the derivation of the mutual information's closed form solution when data is (multivariate-)Gaussian from an existing post and Wikipedia.
And some academic papers say that conditional ...
- 175
2
votes
0
answers
35
views
Maximum Expected Long Term Utility.
I have the following question in Dynamic Games where the first player completely knows the state over all the period $T$ and tries to send signals to the second player (second player only knows the ...
- 31
1
vote
0
answers
21
views
Bounds on $K\left(\mathcal{L}(X_t,X_{t+s})||\mathcal{L}(X_t)\otimes \mathcal{L}(X_{t+s})\right)$ for a diffusion process $(X_t)_{t\geq 0}$?
There are many results giving bounds on the $\beta$-mixing coefficients for diffusion processes (see Proposition 1 in [1] for example). For an homogeneous and stationary process, it means upper ...
- 11
0
votes
0
answers
33
views
How are Shannon's non-physical entropy and physical entropy related?
Suppose there is a die manufacturer. This facility has a dice machine which is in charge of producing new dice by casting their faces in molds made out of some special material, so in a way, it has a ...
- 11
0
votes
0
answers
15
views
Mutual information equals information gain?
There is a data set D. Let the class variable be C and one of the attribute variables be A. Show that the mutual information between the attribute variable A and the class variable C.
The mutual ...
0
votes
1
answer
60
views
Invariance of mutual information under smooth injective mapping
Let $X\colon \Omega\to \mathbb R^m$ and $Y\colon \mathbb \Omega \to\mathbb R^n$ be random variables.
The mutual information is defined as
$$I(X; Y) = \int_{\mathbb R^m\times \mathbb R^n} \log \left( \...
- 3,098
3
votes
1
answer
205
views
The tight bound for conditional mutual information: how much could conditional mutual information be greater than mutual information?
Given random variables $X$,$Y$ and another random variable $Z$, it is known that there are cases when the conditional mutual information $I(X;Y|Z)$ is greater than mutual information $I(X;Y)$.
For ...
- 31
0
votes
1
answer
77
views
Understanding an equality involving Conditional Mutual Information
It is stated in my information theory textbook that the conditional mutual information of (discrete) random variables $X, Y$, given $Z$ is defined as
$I(X; Y|Z) = H(X|Z) - H(X| Y, Z)$, which is equal ...
- 3,738
1
vote
0
answers
41
views
Notation of mutual information for continuous variable
I'm reading a paper, 'Mutual Information Neural Estimator.'
In this paper, the notation of mutual information is written as,
$$ I(X;Z) = \int_{\mathcal{X} \times \mathcal{Z}} \log \frac{d \mathbb{P}_{...
- 11
1
vote
1
answer
49
views
Mutual information of a random subset
Suppose we sample the following two random variables, for some large integer $n$:
Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
- 11
1
vote
0
answers
18
views
Can I(aX+bY,C) larger than max{I(X,C),I(Y,C)}
X,Y are two random variables, and in the range of (0,1)
C is the class tags. C=1,2,3...10. p(C=1)=p(C=2)=p(C=3)....=p(C=10)
a,b are in the range of (0,1)
aX+bY is the linear combination of two random ...
0
votes
0
answers
41
views
Proof of $I(X; Y) = H(Y) - H(Y\vert X)$?
The definition of mutual information between two random variables $X \sim p_{X}$ and $Y \sim p_{Y}$ is given as follows:
$$I(X; Y) := D_{\text{KL}}(p_{X, Y} \ \vert\vert \ p_{X}\otimes p_{Y})$$
I ...
- 642
2
votes
1
answer
178
views
Inequality relating entropy to mutual information
Let $\{X_n\}$ be a sequence of independent, discrete random variables, and let $Z$ be another discrete random variable. Show that $$H(Z)\geq\sum_{i=1}^\infty I(X_i;Z)$$ where $H$ is the entropy and $I$...
- 1,013
0
votes
1
answer
53
views
What are the state-of-the-art dependence measures?
It is well known that mutual information (MI) is a widely-used measure for quantifying statistical dependence between two random variables. Also, I read about other measures such as distance ...
- 400
0
votes
1
answer
59
views
Understanding problem of chain Rule for mutial information
So I have to proof $ I(X;Z|Y) = I(Z;Y|X) - I(Z;Y) + I(X;Z) $
Write mutual information in terms of entropy or use the chain rule for mutual information for an immediate proof.
So I want to directly ...
- 103
3
votes
1
answer
54
views
Stochastic Mutual Information Estimator
I am reading https://openreview.net/forum?id=ByxaUgrFvH and do not understand why they need a "complicated" derivation, because it seems to follow immediately.
Problem
Let $\mathbf{x}$ be a ...
- 269
1
vote
0
answers
42
views
Proof of the information bottleneck equations
In The Information Bottleneck Method, the third term of Eq.(31) is $P_{t+1}(y|\tilde{x})=\sum_yp(y|x)p_t(x|\tilde{x})$, which minimizes the term $D_{KL}[p(y|x)|p(y|\tilde{x})]_{<p(x,\tilde{x})>}$...
- 43
0
votes
0
answers
80
views
Mutual Information of Vectors with Large Inner Product
If we have a joint distribution of two (complex) vectors $x,y\in \mathbb{C}^d$ of norm $1$ such that their inner product $\langle x|y\rangle$ is $1-\epsilon$, can we lower bound the mutual information ...
- 1
0
votes
2
answers
96
views
Why does $H(X|Y)$ equal the "missing information" of $Y$ about $X$?
I've seen mentioned in (Horodecki, Oppenheim, Winter 2005) the fact that the conditional information equals the amount of information that Alice needs to send Bob in order for him to fully reconstruct ...
- 6,483
3
votes
0
answers
75
views
Encoding $I(X;Y)$ into a random variable $Z$ such that $H(Z) = I(X;Y)$ and $I(X;Z) = I(Y;Z) = I(X,Y)$
Is it possible to encode the mutual information $I(X;Y)$ between two random variables $X$ and $Y$ into another random variable $Z$, such that $Z$ "contains" exactly the information that $X$ ...
- 171
0
votes
1
answer
44
views
Mutual Information and Entropy calculation
It is well known that Shannon's joint entropy ($H(X,Y)$) as well as mutual information ($I(X;Y)$) between two variables $X$ and $Y$ are non-negative based on Jensen's inequality. I read in a source ...
- 103
0
votes
0
answers
27
views
comparing mutual information under different constraints
Consider a random variable $X$ taking values in $\{0,1,\ldots,n\}$ and $Y$ takes values in $\{0,1\}$.
Let $a_{i}=P\left(X={i}\right), b_{j}=P(Y=j)$.
Also, $ p_{i}=P\left(Y=0 \mid X=x_{i}\right), q_{i}=...
- 137
0
votes
0
answers
72
views
Maximizing the variance of weights of Bernoulli RV maximize mutual information?
I have a random variable $X=a_1X_1+a_2X_2 + \ldots a_kX_k$ where $X_i \sim Bern(q)$, $X_i \perp X_j, \forall i,j\in \{1,2\ldots,k\}$. Also $\sum_{i=1}^{k} a_i=k$ and $a_i \in \mathbb{N} \bigcup \{0\...
- 137
1
vote
0
answers
75
views
Correlations in implementations of random unitary channels
A random unitary channel (RUC) acting on a quantum system $S$ is any completely positive and trace preserving (CPTP) linear map $\mathcal{E}$ that can be expressed as
\begin{align}
\mathcal{E}(\...
- 11
4
votes
0
answers
116
views
memory increases channel capacity?
I learnt in $\textit{Elements of Information Theory -Cover & Thomas}$ that $\underline{\text{memory increases the capacity of a channel}}$.
A transmitter sends $X_i$ and the receiver receives $...
- 137
3
votes
2
answers
168
views
Entropy of (a binary random variable plus gaussian random variable) increases with distance between binary values
Let $X\in\{x_1,x_2\}$ be a real binary random variable with probability mass function $\{p_1,p_2=1-p_1\}$. Let $N$ be a real random variable with standard normal distribution (mean = $0$, std = $1$). ...
- 51
0
votes
1
answer
24
views
why this conditional entropy is equal $H(y|x) = H(y-f(x) |x)$
I was trying to understand the equation 8.11 to 8.12 of this paper, where 8.11 suggested:
$H(x, y) = H(x) + H((y|x) = H(x) + H(y - f(x)|x) = H(x) + H(n|x) =H(x,n)$
After searching, I can't find any ...
- 1
1
vote
1
answer
394
views
How can mutual information between random variables decrease with increasing correlation?
I have a problem where the mutual information between dependent random variables seems to decrease as their correlation increase, which goes against my intuition of mutual information. I'd appreciate ...
- 155
6
votes
1
answer
323
views
Mutual information is maximized when Poisson-Binomial reduces to Binomial?
I have two random variables $X$ and $Y$. $X$ follows Poisson-Binomial distribution with parameters $\{q_1, \ldots, q_k\}$. Thus, $X$ can take values in the set $\{0,1,\ldots,k\}$.
$Y$ is a binary ...
- 137
2
votes
0
answers
45
views
Difference of KL Divergences
I am interested in the difference of KL divergences
\begin{align}
KL&(p(x)||q(x))-KL(p(x|y)||q(x|y)) = \\ &\int p(x)\log\dfrac{p(x)}{q(x)}dx-\int p(y)\int p(x|y)\log\dfrac{p(x|y)}{q(x|y)}dxdy
\...
- 150
0
votes
0
answers
75
views
How to minimize mutual information between a variable and a correlated sequence of variables?
I'm working on a problem of minimizing mutual information between a variable and a correlated sequence of variables.
Formally, I have random variable $Y$ and a sequence of random variables $X_{1}, X_{...
- 1