Skip to main content

Questions tagged [mutual-information]

Mutual Information is a metric used in Information Theory. It describes the amount of information shared by two random variables. Extensions for larger number of variables exist. See https://en.wikipedia.org/wiki/Mutual_information

Filter by
Sorted by
Tagged with
0 votes
1 answer
51 views

Mutual information expansion not justifiable

I have recently read a mutual information term, $I(X;Y,Z)=E_{p(X,Y,Z)}\big[\log\frac{ p(X|Y,Z)}{p(X)}\big]$. While this expansion does not make sense to me. Is it correct? My understanding (using ...
Aleph's user avatar
  • 164
0 votes
1 answer
31 views

Mutual information between randomly permuted variables

I am trying to understand how to calculate the mutual information between two variables which have nonzero conditional mutual information (when conditioned on a third variable) but (I believe) zero ...
asph's user avatar
  • 406
0 votes
0 answers
29 views

How to find "extra" mutual information

Codewords $00$ and $11$ are sent with equal probability through a BSC with error probability p. Compute the mutual information between the codeword sent and the first digit received as output. I have ...
S. Chitratta's user avatar
2 votes
0 answers
71 views

Mutual information when sampling a random variable multiple times

Let $X$ be a random variable. For a fixed (known) preparation of $X$, suppose I have a protocol that generates a second random variable, $Y$, in a way that indirectly depends on $X$. Ultimately the ...
smitke6's user avatar
  • 699
0 votes
1 answer
36 views

Estimating the conditional entropy of a discrete variable conditioning on continuous variable

I am doing a machine learning project and I am trying to select the best features by computing their mutual information and select the ones with the highest information gain. I was looking at this ...
Ishigami's user avatar
  • 1,717
0 votes
0 answers
23 views

Maximum Capacity of a Communication Channel w.r.t. $P(Y | X)$ when $X$ and $Y$ are discrete

Let $X$ and $Y$ be two discrete stochastic variables. I want to find $P(Y|X)$ that maximizes the mutual information between $X$ and $Y$, i.e., $$\max_{P(Y|X)} I(X, Y).$$ This problem is different from ...
Sam's user avatar
  • 357
0 votes
0 answers
32 views

Mutual Information of two conditionally independent varibales given other varibale

I have data as $X$ which has two independent labels as $Y$ and $M$. So the goal is to process the $X$ to $Z_1=\mathcal{F}_1(X)$ to maximize $I(Z_1;Y|M)$. We know that: $$I(Z_1;Y|M) = I(Z_1;Y) - I (Z_1;...
mahdi_12167's user avatar
0 votes
1 answer
93 views

Negative mutual information example. What's wrong about it?

I'm aware that by definition the Mutual Information (MI) should be non-negative, and there are two related questions here: (1) and (2). However, I can think of an example in Physics where it is (or at ...
Girardi's user avatar
  • 355
1 vote
1 answer
40 views

What is the relationship between mutual information conditioned on different variables: I(W;X|Y) vs. I(W;X|Z)

Let four random variables form the Markov chain $${\displaystyle \raise{1.5ex}W\overset{\longleftarrow}{\searrow\lower{1.5ex} X\swarrow}\raise{1.5ex} Y\searrow\lower{1.5ex} Z}$$ such that the ...
Jabby's user avatar
  • 177
0 votes
0 answers
33 views

Showing that the claim $I(X;Y) = I(X;Z)= 0 \implies I(X;Y;Z) = 0$ is true for jointly normal random variables only.

If we have three random variables $X,Y,Z$, which are jointly normal, how can it be shown that $I(X;Y) = I(X;Z)= 0 \implies I(X;Y;Z) = 0$? I know that for jointly normal distributions $X,Y,Z$: $I(X;Y) =...
proton100's user avatar
1 vote
0 answers
22 views

How to construct a maximum-information embedding of sampled objects using a binary function?

This feels like a very specific problem, but I hope there already is a method to achieve what I want. There is a random process from which I can draw samples of non-numerical, variable sized objects (...
fazekaszs's user avatar
  • 153
1 vote
1 answer
51 views

mutual information of two normal random vectors

I'm dealing with a question that request to calculate the mutual information of two normal random vectors, this is the description: If $\mathbf X\sim \mathcal N(\mu_X,\Sigma_X),~\mathbf Y\sim \mathcal ...
ShuchenSean's user avatar
0 votes
1 answer
35 views

Mutual information analysis for sum of Bernoulli and Gaussian

Question: I have two random variables $X\sim\text{Bernoulli}(\alpha)$ and $Y\sim\mathcal{N}(0,1)$. Is it possible to compute the mutual information $I(X;X+Y)$ analytically? My attempt: By the ...
Resu's user avatar
  • 816
3 votes
1 answer
66 views

Derivative of Discrete Mutual Information

Given a fixed channel $p(y|x)$, and denoting the discrete input pmf $p(x)$, and the corresponding output pmf by $p(y)$, prove that: $$ \frac{\partial I(X;Y)}{\partial p(x)} = I(X=x;Y) - \log{e} ...
Math Lover's user avatar
0 votes
0 answers
36 views

interpretation of negative pointwise mutual information

Since: $$pmi(x,y)\quad =\quad \log(\frac{P(X=x,Y=y)}{P(X=x)P(Y=y)})\quad =\quad -t$$ iff $$ 2^{t}P(Y=y | X=x) \quad =\quad P(Y=y)$$ the interpretation of a negative PMI seems very clear to me. So I ...
kuemmel's user avatar
  • 73
0 votes
0 answers
35 views

Maximizing the expectation in CE Importance sampling.

Suppose the following maximization: $$v_t = \arg \max_{v} E_{v_{t-1}} 1\{S(x) \geq \gamma\} \frac{f(x;u)}{f(x;v_{t-1})}\ln f(x;v) = max_{v} E_{v_{t-1}} 1\{S(x) \geq \gamma\} W(u;v_{t-1}) \ln f(x;v),$$ ...
entropy's user avatar
  • 147
1 vote
0 answers
25 views

Inequality related to cross-enropy

Suppose, $S: x \mapsto \mathbb{R}^1$ and $f(\cdot;p)$ is some parametric family of pdfs. Moreover, $\sum_{x}1 \{S(X) \geq \gamma \} f(x;u)\log f(x;v) \geq \sum_{x}1\{S(X)\geq \gamma\}f(x;u)\log f(x;u)...
entropy's user avatar
  • 147
0 votes
0 answers
37 views

Mutual information between standard gaussian and bernoulli distribution

I have a random variable which is distribuited as a multinomial standard gaussian $z\sim N^d(0, I)$, and another r.v. that instead is distribuited as a bernoulli $\alpha \sim Bern(\pi)$. Is it ...
Francesco De Santis's user avatar
0 votes
1 answer
48 views

Apparent contradiction involving differential mutual information

Suppose X is a continuous random variable, and Y = 2X is a deterministic transformation of it. Then the following identities hold, according to Cover & Thomas: $$ I(X; Y) = h(X) - h(X|Y) \\ I(X; Y)...
canjobear's user avatar
2 votes
0 answers
43 views

Equvivalent definitions of mutual information of continuous random variables

I am reading Elements of Information Theory by Cover and Thomas (2006) and struggle with the definition of mutual information for continuous random variables (Chapter 9: Differential Entropy). For two ...
Avec's user avatar
  • 21
0 votes
0 answers
14 views

Find maximum likelihood estimator of joint probability distribution of Bernoulli variables (for use in mutual information/feature selection)

When performing feature selection by finding mutual information estimates for class C and feature U (both binary), we need to estimate joint probabilities like P(C=1, U=1). This site claims that the ...
user156220's user avatar
1 vote
1 answer
91 views

Determine the channel capacity of binary DMC

I'm trying to determine the channel capacity of the following DMC DMC Converting this into a PTM, I get the following PTM = [[1,0,0], [1/3,1/3,1/3], [0,0,1]] I can see this is not symmetric or weakly ...
FoxMaccloud's user avatar
1 vote
0 answers
42 views

Applying Blahut-Arimoto Algorithm for Continuous Channels Capacity Calculation

I am exploring the Blahut-Arimoto algorithm for channel capacity calculations, particularly for continuous channels. I understand its application in discrete scenarios, but I am unsure about its ...
Alireza Ghazavi's user avatar
0 votes
0 answers
45 views

Challenges in Implementing the Blahut–Arimoto Algorithm for channel capacity computation

I've read the Blahut–Arimoto Algorithm from the original paper by Blahut and Raymond Yeung's book, along with other sources. However, I'm having trouble understanding the computation of channel ...
Alireza Ghazavi's user avatar
1 vote
1 answer
58 views

likelihood and fisher information of two dependent variables

Let $\{X, Y\}$ be two sets of variables that depend on the parameter $\theta$. Let $z_1 = f_1(X,Y), z_2 = f_2(X,Y)$ be two variables constructed from $\{X, Y\}$. The functions $f_i$'s are known. $I_{...
lost-neutrino's user avatar
2 votes
0 answers
73 views

Does Mutual information capture both positive and negative dependence equally?

Consider the following formula for mutual information (MI) between continuous random variables X and Y: $$I(X; Y) = \iint f(x,y) \log\left(\frac{f(x,y)}{g(x)h(y)}\right) \, dx \, dy$$ I 've read ...
Enk9456's user avatar
  • 131
1 vote
1 answer
52 views

"Random" Function of a Random Variable (Cover & Thomas Elements of Information Theory)

I'm looking to clarify a technical point about 'random processing' in Cover & Thomas. They've occasionally refer to a function of a random variable as being 'allowed to be random.' Here are some ...
Josh Keneda's user avatar
  • 2,967
0 votes
0 answers
21 views

Mutual information of the joint distribution and the marginal distribution if two random variables are independent?

if $Y \rightarrow X$ and $Y \rightarrow Z$, is it possible to find the equality/inequality between $$ I(X,Z; Y) $$ and $$ I(X;Y) + I(Z;Y) $$ Or equivently, given $I(X,Z;Y)= I(X; Y) + I (Z; Y |X)$, ...
qiuweikang's user avatar
0 votes
1 answer
244 views

Consider a fair coin flip. What is the mutual information between the top side and the bottom side of the coin?

Mutual information between the top side and the bottom side of the coin?. T is the top side, B is the bottom side. $$I(T;B) = H(B) - H(B|T) = \log(2) = 1$$ the log base is 2. I don't know why the ...
Jim's user avatar
  • 3
0 votes
1 answer
106 views

Is Entropy of a Discrete Random Variable Condition on a Continuous Random Variable Always bigger than or Equal to Zero? [closed]

Let $X$ be a discrete random variable and $Y$ be a continuous random variable. Is the conditional entropy of $X$ given $Y$ always positive, i.e., $H(X|Y)\ge0$?
Nick Cooper's user avatar
0 votes
0 answers
18 views

Finding mutual information in discrete linear partial observation stochastic process

I have one basic question maybe is not to hard for you but I am a bit confused. Let our system be like this: \begin{align} X_{k+1} &= A_k X_k + W_k \\ Y_k &= C_k X_k + V_k \end{align} where $...
Kasi's user avatar
  • 63
0 votes
0 answers
296 views

$f$-mutual information and chain rule

The standard definition of mutual information for a pair of random variables $X, Y$ is $I(X; Y) = D_{KL}(P_{X, Y} \| P_X \times P_Y)$. One of the most important properties of this definition is the ...
susami's user avatar
  • 311
1 vote
0 answers
44 views

Mutual information of dependent variables

I am looking at a setting where I have a random variable $X$ and two other random variables that are derived from $X$, $Y = f(X)$ and $Z =g(X)$. What I want to do is compute the mutual information ...
ac1643's user avatar
  • 57
0 votes
0 answers
36 views

Characterization of functions that have the data processing inequality

I wonder if people have studied the characterization of data processing inequality. Here's the data processing inequality for mutual information: 3 variables for Markov chain: $X\to Y\to Z$, then $I(X,...
Evelyn000's user avatar
1 vote
0 answers
144 views

What is the mutual information between ingredients of a mixture of Gaussians?

Consider 2 random variables, $X$ and $Y$. $X$ is discrete and $Y$ is continuous. In particular, we have a Gaussian distribution $Y$ with mean $X$ and variance $\sigma = 1$, and $P(X=1) = \frac{1}{4}$...
Daniel S.'s user avatar
  • 853
1 vote
1 answer
51 views

Total Correlation is difference of relative entropies in general?

Motivation Consider a finite set $[q]=\{1,\dots,q\}$, random variables $X_1,\dots,X_k\in[q]$, and their product $X=X_1\otimes\cdots\otimes X_k\in[q]^k$, i.e. the components of $X$ are independent. Let ...
Matija's user avatar
  • 3,623
0 votes
1 answer
91 views

Intuition on Mutual information in XAI

Following wikipedia, mutual information $MI(X,Y)$ is 'how much knowledge is in the output of one random variable about the other variable'. If MI between 2 variables is high, this does not mean I know ...
Mathematiger's user avatar
0 votes
0 answers
61 views

Derivative of a mutual information for a Gaussian channel

I saw this question which inspired me to ask mine. Given that I have a mutual information $$ I\left(X;\sqrt{\frac{1}{(x/\delta)+\sigma^2}}\cdot X + Y\right), $$ which is a function of $x$ that is not ...
Resu's user avatar
  • 816
2 votes
2 answers
52 views

Why does the common information $C\equiv X\wedge Y$ give $H(X)=H(CX)$?

Given two random variables $X,Y$, their common information $X\wedge Y$ is defined in (Wolf and Wultschleger 2004) as the random variable $X\wedge Y=f_X(X)=f_Y(Y)$ constructed as follows: Let $G$ be ...
glS's user avatar
  • 7,095
8 votes
1 answer
1k views

Is there a chain rule for Sibson's mutual information?

Mutual Information satisfies the chain rule: $$I(X,Y;Z) = I(X;Z) + I(Y;Z|X).$$ The chain rule is useful and the proof is simply linearity of expectations. Sometimes we want something stronger than ...
Thomas's user avatar
  • 915
2 votes
0 answers
42 views

Maximum Expected Long Term Utility.

I have the following question in Dynamic Games where the first player completely knows the state over all the period $T$ and tries to send signals to the second player (second player only knows the ...
Zeidan Braik's user avatar
1 vote
0 answers
28 views

Bounds on $K\left(\mathcal{L}(X_t,X_{t+s})||\mathcal{L}(X_t)\otimes \mathcal{L}(X_{t+s})\right)$ for a diffusion process $(X_t)_{t\geq 0}$?

There are many results giving bounds on the $\beta$-mixing coefficients for diffusion processes (see Proposition 1 in [1] for example). For an homogeneous and stationary process, it means upper ...
Zini's user avatar
  • 11
0 votes
1 answer
96 views

Invariance of mutual information under smooth injective mapping

Let $X\colon \Omega\to \mathbb R^m$ and $Y\colon \mathbb \Omega \to\mathbb R^n$ be random variables. The mutual information is defined as $$I(X; Y) = \int_{\mathbb R^m\times \mathbb R^n} \log \left( \...
Paweł Czyż's user avatar
  • 3,320
3 votes
1 answer
347 views

The tight bound for conditional mutual information: how much could conditional mutual information be greater than mutual information?

Given random variables $X$,$Y$ and another random variable $Z$, it is known that there are cases when the conditional mutual information $I(X;Y|Z)$ is greater than mutual information $I(X;Y)$. For ...
Jiayun Wu's user avatar
0 votes
1 answer
101 views

Understanding an equality involving Conditional Mutual Information

It is stated in my information theory textbook that the conditional mutual information of (discrete) random variables $X, Y$, given $Z$ is defined as $I(X; Y|Z) = H(X|Z) - H(X| Y, Z)$, which is equal ...
IntegrateThis's user avatar
1 vote
0 answers
63 views

Notation of mutual information for continuous variable

I'm reading a paper, 'Mutual Information Neural Estimator.' In this paper, the notation of mutual information is written as, $$ I(X;Z) = \int_{\mathcal{X} \times \mathcal{Z}} \log \frac{d \mathbb{P}_{...
Hoin's user avatar
  • 11
1 vote
1 answer
54 views

Mutual information of a random subset

Suppose we sample the following two random variables, for some large integer $n$: Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
LearningMath's user avatar
1 vote
0 answers
20 views

Can I(aX+bY,C) larger than max{I(X,C),I(Y,C)}

X,Y are two random variables, and in the range of (0,1) C is the class tags. C=1,2,3...10. p(C=1)=p(C=2)=p(C=3)....=p(C=10) a,b are in the range of (0,1) aX+bY is the linear combination of two random ...
BBBBYYYYYKKKKK's user avatar
0 votes
0 answers
41 views

Proof of $I(X; Y) = H(Y) - H(Y\vert X)$?

The definition of mutual information between two random variables $X \sim p_{X}$ and $Y \sim p_{Y}$ is given as follows: $$I(X; Y) := D_{\text{KL}}(p_{X, Y} \ \vert\vert \ p_{X}\otimes p_{Y})$$ I ...
Hermi's user avatar
  • 702
2 votes
1 answer
232 views

Inequality relating entropy to mutual information

Let $\{X_n\}$ be a sequence of independent, discrete random variables, and let $Z$ be another discrete random variable. Show that $$H(Z)\geq\sum_{i=1}^\infty I(X_i;Z)$$ where $H$ is the entropy and $I$...
verygoodbloke's user avatar