Questions tagged [mutual-information]

Mutual Information is a metric used in Information Theory. It describes the amount of information shared by two random variables. Extensions for larger number of variables exist. See https://en.wikipedia.org/wiki/Mutual_information

Filter by
Sorted by
Tagged with
-1 votes
0 answers
51 views

Chain rule independent variables

Does the following identity $$ I(X_1,X_2 ; Y_1, Y_2) = I(X_1; Y_1) + I(X_2; Y_2) $$ hold for mutual information for $(X_1, Y_1)$ and $(X_2, Y_2)$ independent? Attempt: $p(x_1, x_2, y_1, y_2) = p(x_1, ...
user avatar
  • 322
0 votes
0 answers
16 views

Mutual Information between $v_1$ and $v_2$ coming from the same Inverse-Wishart distribution?

Say that $\left(\begin{matrix} v_1 & c\\ c & v_2 \end{matrix}\right)$ is a bivariate covariance matrix that comes from an Inverse-Wishart distribution $W^{-1}(\Psi, \nu)$. Then what is the ...
user avatar
  • 1,642
2 votes
1 answer
40 views

Inequality relating entropy to mutual information

Let $\{X_n\}$ be a sequence of independent, discrete random variables, and let $Z$ be another discrete random variable. Show that $$H(Z)\geq\sum_{i=1}^\infty I(X_i;Z)$$ where $H$ is the entropy and $I$...
user avatar
0 votes
1 answer
35 views

What are the state-of-the-art dependence measures?

It is well known that mutual information (MI) is a widely-used measure for quantifying statistical dependence between two random variables. Also, I read about other measures such as distance ...
user avatar
  • 298
0 votes
1 answer
22 views

Understanding problem of chain Rule for mutial information

So I have to proof $ I(X;Z|Y) = I(Z;Y|X) - I(Z;Y) + I(X;Z) $ Write mutual information in terms of entropy or use the chain rule for mutual information for an immediate proof. So I want to directly ...
user avatar
  • 103
3 votes
1 answer
41 views

Stochastic Mutual Information Estimator

I am reading https://openreview.net/forum?id=ByxaUgrFvH and do not understand why they need a "complicated" derivation, because it seems to follow immediately. Problem Let $\mathbf{x}$ be a ...
user avatar
0 votes
0 answers
19 views

Proof of the information bottleneck equations

In The Information Bottleneck Method, the third term of Eq.(31) is $P_{t+1}(y|\tilde{x})=\sum_yp(y|x)p_t(x|\tilde{x})$, which minimizes the term $D_{KL}[p(y|x)|p(y|\tilde{x})]_{<p(x,\tilde{x})>}$...
user avatar
0 votes
0 answers
32 views

Mutual Information of Vectors with Large Inner Product

If we have a joint distribution of two (complex) vectors $x,y\in \mathbb{C}^d$ of norm $1$ such that their inner product $\langle x|y\rangle$ is $1-\epsilon$, can we lower bound the mutual information ...
user avatar
  • 1
0 votes
0 answers
29 views

Mutual information as supermodular function under independence?

Let $Y$ be a response variable and $F$ be a set of features such that $f,x \in F$ and $S \subset F$. I am interested in the difference of mutual informations and would like to show the following ...
user avatar
  • 305
0 votes
2 answers
74 views

Why does $H(X|Y)$ equal the "missing information" of $Y$ about $X$?

I've seen mentioned in (Horodecki, Oppenheim, Winter 2005) the fact that the conditional information equals the amount of information that Alice needs to send Bob in order for him to fully reconstruct ...
user avatar
  • 5,565
3 votes
0 answers
55 views

Encoding $I(X;Y)$ into a random variable $Z$ such that $H(Z) = I(X;Y)$ and $I(X;Z) = I(Y;Z) = I(X,Y)$

Is it possible to encode the mutual information $I(X;Y)$ between two random variables $X$ and $Y$ into another random variable $Z$, such that $Z$ "contains" exactly the information that $X$ ...
user avatar
  • 171
0 votes
1 answer
32 views

Mutual Information and Entropy calculation

It is well known that Shannon's joint entropy ($H(X,Y)$) as well as mutual information ($I(X;Y)$) between two variables $X$ and $Y$ are non-negative based on Jensen's inequality. I read in a source ...
user avatar
0 votes
0 answers
23 views

comparing mutual information under different constraints

Consider a random variable $X$ taking values in $\{0,1,\ldots,n\}$ and $Y$ takes values in $\{0,1\}$. Let $a_{i}=P\left(X={i}\right), b_{j}=P(Y=j)$. Also, $ p_{i}=P\left(Y=0 \mid X=x_{i}\right), q_{i}=...
user avatar
0 votes
0 answers
15 views

Is my intuition about the differential entropy and mutual information correct

If $X$ is a well-behaved continuous random variable, is it true that $$H(XX) = H(X)$$ $$I(X:X) = H(X)$$ This is certainly true for discrete variables, since (assuming X = Y) $$H(XY) = H(X|Y) + H(Y) = ...
user avatar
0 votes
0 answers
69 views

Maximizing the variance of weights of Bernoulli RV maximize mutual information?

I have a random variable $X=a_1X_1+a_2X_2 + \ldots a_kX_k$ where $X_i \sim Bern(q)$, $X_i \perp X_j, \forall i,j\in \{1,2\ldots,k\}$. Also $\sum_{i=1}^{k} a_i=k$ and $a_i \in \mathbb{N} \bigcup \{0\...
user avatar
1 vote
0 answers
56 views

Correlations in implementations of random unitary channels

A random unitary channel (RUC) acting on a quantum system $S$ is any completely positive and trace preserving (CPTP) linear map $\mathcal{E}$ that can be expressed as \begin{align} \mathcal{E}(\...
user avatar
  • 11
4 votes
0 answers
72 views

memory increases channel capacity?

I learnt in $\textit{Elements of Information Theory -Cover & Thomas}$ that $\underline{\text{memory increases the capacity of a channel}}$. A transmitter sends $X_i$ and the receiver receives $...
user avatar
2 votes
2 answers
87 views

Entropy of (a binary random variable plus gaussian random variable) increases with distance between binary values

Let $X\in\{x_1,x_2\}$ be a real binary random variable with probability mass function $\{p_1,p_2=1-p_1\}$. Let $N$ be a real random variable with standard normal distribution (mean = $0$, std = $1$). ...
user avatar
  • 31
0 votes
1 answer
19 views

why this conditional entropy is equal $H(y|x) = H(y-f(x) |x)$

I was trying to understand the equation 8.11 to 8.12 of this paper, where 8.11 suggested: $H(x, y) = H(x) + H((y|x) = H(x) + H(y - f(x)|x) = H(x) + H(n|x) =H(x,n)$ After searching, I can't find any ...
user avatar
  • 1
1 vote
1 answer
105 views

How can mutual information between random variables decrease with increasing correlation?

I have a problem where the mutual information between dependent random variables seems to decrease as their correlation increase, which goes against my intuition of mutual information. I'd appreciate ...
user avatar
  • 155
6 votes
1 answer
146 views

Mutual information is maximized when Poisson-Binomial reduces to Binomial?

I have two random variables $X$ and $Y$. $X$ follows Poisson-Binomial distribution with parameters $\{q_1, \ldots, q_k\}$. Thus, $X$ can take values in the set $\{0,1,\ldots,k\}$. $Y$ is a binary ...
user avatar
2 votes
0 answers
39 views

Difference of KL Divergences

I am interested in the difference of KL divergences \begin{align} KL&(p(x)||q(x))-KL(p(x|y)||q(x|y)) = \\ &\int p(x)\log\dfrac{p(x)}{q(x)}dx-\int p(y)\int p(x|y)\log\dfrac{p(x|y)}{q(x|y)}dxdy \...
user avatar
  • 140
0 votes
0 answers
31 views

Order of correction term of density estimation

I am trying to follow along with the paper Estimating Mutual Information by Kraskov et. al. (2008). Equation 19 in this paper is an estimation of the log density of the data. What I am uncertain of is ...
user avatar
  • 11
0 votes
0 answers
8 views

Clustering Similarity Measurement Based on Mutual Information

I have a question when learning the measures based on mutual information to the similarity between two clusterings. See this paper. Say the labels can be permuted. For example, "both a and b ...
user avatar
  • 1,466
0 votes
0 answers
47 views

How to minimize mutual information between a variable and a correlated sequence of variables?

I'm working on a problem of minimizing mutual information between a variable and a correlated sequence of variables. Formally, I have random variable $Y$ and a sequence of random variables $X_{1}, X_{...
user avatar
0 votes
1 answer
43 views

Data Processing Inequality for Random Vectors forming a Markov Chain

Suppose 3 $n$-length random vectors form a Markov chain: $\mathbf{X} \rightarrow \mathbf{Y} \rightarrow \mathbf{Z}$. Moreover, $Y_{i}$ is a deterministic function of $Z_{i}$ (ie. $Y_{i}=g_{i}(Z_{i})$)....
user avatar
  • 27
0 votes
1 answer
40 views

Find the parameter minimizing KL divergence

For discrete probability distributions $P$ and $Q$ defined on the same probability space, $\mathcal{X}$, the relative entropy from $Q$ to $P$ is defined to be $$ D_{\mathrm{KL}}(P \| Q)=\sum_{x \in \...
user avatar
0 votes
1 answer
62 views

Prove that something that can be learned from $S=\sum_{i=1}^n x_i$ is not less than what can be learned from $S+x_{n+1}$

Is there a way to formally prove that whatever can be learned about any $x_i$ from $S=\sum_{i=1}^n x_i$ is certainly not less than what can be learned from $S'=S+x_{n+1}$ where $x_i$s belong to some ...
user avatar
3 votes
0 answers
69 views

Capacity of a 2-user discrete memoryless channel

I am trying to understand the capacity of the following 2 user multiple access channel with input and output alphabets $\mathcal{X}_{1}=\mathcal{X}_{2}=\mathcal{Y}=\{0,1\}$. The channel transition ...
user avatar
1 vote
1 answer
25 views

Capacity of a discrete memoryless channel with infinite input alphabet size

I have been reading "Elements of Information Theory"- Cover & Thomas. It states the following: Definition: A discrete channel, denoted by $(\mathscr{X}, p(y \mid x), \mathscr{Y})$, ...
user avatar
0 votes
0 answers
20 views

Multivariate normal distribution; joint entropy

I have data that is the combination of two classes of multivariate normal distribution. I have used em algorithm to get mean, standard derivation, and weights for them. Could anyone let me know how to ...
user avatar
0 votes
0 answers
37 views

Basic Fundamental Information Theory Question. Unlikely events give more information.

Hi I have a really basic question about information theory : Reading Pattern Recognition and Machine Learning by Christopher Bishop he says ( in chapter 1.6) "If we are told that a highly ...
user avatar
  • 1,778
0 votes
0 answers
21 views

If the diagonal constraints were considered using conventional waterfilling, what can we do?

$$\begin{aligned} &\max _{\mathbf{\mathbf{R}_\mathrm{X}}} \log _{2} \operatorname{det}\left(I_{N_r}+\frac{a}{N_r} \boldsymbol{A}_{\mathrm{R}} \boldsymbol{H}_{\mathrm{V}}\mathbf{R_\mathrm{X}} \...
user avatar
1 vote
0 answers
18 views

Estimating mutual information from local densities

Consider the two multivariate random variables $X$ and $Y$. Their mutual information (MI) is defined as $$\begin{aligned} I(X;Y) &\stackrel{\mathrm{def}}{=} D_\mathrm{KL}(P_{(X,Y)}||P_{X}\otimes ...
user avatar
  • 111
2 votes
1 answer
70 views

Maximizing Mutual information of a Fading Channel with Additive Gaussian Noise

I am considering a problem such that we have a channel $Y=XV+Z$. Assuming $X,V,Z$ all independent Where $$Z \sim \mathcal{N}(0, N)$$ and $$ V=\begin{cases} \alpha_1, & p\\ \alpha_2, & (1-p) \...
user avatar
  • 357
1 vote
1 answer
54 views

Calculating Capacity of a channel with mixture distribution.

Given a channel $Y=X+Z$ and a random variable $Z$ such that $$ Z=\begin{cases} 0, &w.p. \frac{1}{10}\\ Z^* &w.p. \frac{9}{10} \end{cases} $$ such that $Z^* \sim \mathcal{N}(0,\sigma^2)$ What ...
user avatar
  • 357
1 vote
1 answer
35 views

Can the joint entropy be expressed in terms of pairwise measures?

Assume we have $N$ random variables $x_1, x_2, \dots, x_N$. Is there a way to express the joint entropy $H(x_1, x_2, \dots, x_N)$ in terms of single-variable or pairwise measures such as the pairwise ...
user avatar
  • 135
0 votes
1 answer
33 views

Conditional Mutual Information - Cover exercise

Following is the exercise from T. Cover's Elements of Information Theory While I understand that $$I(X_{n-1}; X_n \mid X_1, \ldots, X_{n-2}) = H(X_{n-1} \mid X_1, \ldots, X_{n-2}) - H(X_{n} \mid X_1, ...
user avatar
1 vote
0 answers
44 views

Can information transmission be proven in a Rule 30 ECA?

(This is hopefully a clearer version of an earlier post of mine.) I have been spending lots of time on the open challenge of proving the aperiodicity of the central column of a rule 30 cellular ...
user avatar
  • 4,979
0 votes
0 answers
145 views

Mutual Information for Continuous Random variables for system with 1 input and multiple outputs (SIMO System)

Let's consider channels of the form \begin{equation}\label{eq:channelmodel2} Y_{1}= X e^{j \theta_{1}}+ V_1 \end{equation} \begin{equation}\label{eq:channelmodel3} Y_{2}=X e^{j \theta_{2}}+ V_2 \end{...
user avatar
  • 1
1 vote
0 answers
65 views

prove $I(X_1;X_2) \ge I(F_1(X_1);F_2(X_2))$

Suppose $X_1,X_2$ are random variables. Discrete. Also, assume $Y_1 =F_1(X_1), Y_2=F_2(X_2)$. Prove the following relation. $$I(X_1;X_2) \ge I(Y_1;Y_2)$$ I think the solution like this. Is it right? ...
user avatar
  • 435
0 votes
0 answers
46 views

Geometric Interpretation of negativity of interaction information

The interaction information (II) in 3D is defined as a generalization of the 2D mutual information (MI). $$ \begin{eqnarray} I(X:Y:Z) &=& I(X:Y) - I(X:Y|Z) \\ &=& H(X) + H(Y)+H(Z) - H(...
user avatar
15 votes
2 answers
14k views

What is the simplest proof that the mutual information $I(X:Y)$ is always non-negative?

What is the simplest proof that mutual information is always non-negative? i.e., $I(X;Y)\ge0$
user avatar
  • 639