0
votes
0answers
46 views

Special case of Kullback-Leibler additivity

I have three random variables $X,Y,Z$. If $(X,Z)$ are an independent pair and $(Y,Z)$ are an independent pair, then the additive property of the Kullback-Leibler divergence says $K(X,Z|Y,Z) = K(X|Y) ...
3
votes
1answer
49 views

Upper Bound on Mutual Information

I am interested in an upper bound on mutual information that I have been encountering frequently in the statistics and probability literature. I have yet to see the "purest" form of the inequality, so ...
0
votes
1answer
34 views

A question on Markov chain

Suppose for two random variables $X$ and $Y$ we have $X\perp\!\!\!\perp Y$ and also assume that three random variables $X$, $Y$ and $Z$ form the following Markov chain: $X\to Z\to Y$. Do these two ...
0
votes
1answer
19 views

Loss of information while projecting multidimensional data

I'm interested in the evaluation of the loss of information after projecting multidimensional data. Since the dimensional reduction is a common tool to analyse data,a question about the loss of ...
1
vote
1answer
117 views

The golden ratio in statistics of literature

Let a book, for example, or a poem... It consists in words and letters and symbols like : ;,!... Let $W_b$=the number of words of the book. Let $L_b$=the number of letters of the book. The number ...
4
votes
1answer
53 views

Joint Probability from Marginal Probabilities

$X, Y_1, Y_2$ are random variables with (possibly) different finite alphabets. For given conditional probability mass functions $\mathbb{P}(Y_1|X)$ and $\mathbb{P}(Y_2|X)$, is it always possible to ...
2
votes
3answers
245 views

Is Standard Deviation the same as Entropy?

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution ...
2
votes
1answer
37 views

Is it true that $H(X|Y)=H(Y|X)$?

I have some difficulties with the question whether $H(X|Y)=H(Y|X)$? From my knowledge $I(X;Y)=H(X)-H(X|Y) = H(Y)-H(Y|X)$ so $H(X|Y)=H(Y|X)$ only when $H(X)=H(Y)$ The question is whether it's ...
3
votes
2answers
130 views

Estimating the entropy

Given a discrete random variable $X$, I would like to estimate the entropy of $Y=f(X)$ by sampling. I can sample uniformly from $X$. The samples are just random vectors of length $n$ where the entries ...
0
votes
2answers
49 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
2
votes
1answer
63 views

Confused about notation: difference between $\prod_{i=1}^np(x_i)$ and $\prod_{i=1}^np(x)$

In my information theory book by Cover and Thomas, at the beginning of the channel coding theorem, it's written: "Each entry in this matrix" (the matrix of the randomly generated code) "is ...
0
votes
1answer
151 views

If $f_\theta=Uniform(\theta,\theta +1)$, a sufficient statistic for $\theta$ is… but why?

If $f_\theta=\mathrm{Uniform}(\theta,\theta +1)$, a sufficient statistic for $\theta$ is $$T(X_1,X_2,\dots,X_n)=(\max\lbrace X_1,X_2,\dots,X_n\rbrace,\min\lbrace X_1,X_2,\dots,X_n\rbrace).$$ Can ...
3
votes
2answers
167 views

Markov chains for beginners, how to think about them?

So this is what my book states: Random variables $X,Y, and Z$ are said to form a Markov chain in that order denoted $X\rightarrow Y \rightarrow Z$ if and only if: $p(x,y,z)=p(x)p(y|x)p(z|y) $ ...
2
votes
2answers
250 views

Random process, stochastic process explained intuitively?

So I've read the definitions online and this is what I understood. $X(t)$ is a random process for $t>0$ and we can think of it as being a random variable at any given time $t=t_0$. For example, ...
1
vote
2answers
285 views

“by definition A and B R.V are independent means that: $p(A∪B)=p(A)+p(B)$ right?” No, absolutely not right.

Can someone please explain why? Isn't $p(a,b)=p(a)*p(b) $ equivalent to $p(A∪B)=p(A)+p(B)$? If not can you please give a counterexample or something? Thanks a lot!
2
votes
1answer
87 views

I.I.D what does this stand for?

So almost everywhere in the book it's written "random variables are IID", what does this mean? I think it means independent and identically distributed but not sure. So by definition A and B R.V are ...
1
vote
1answer
37 views

Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$

Let $X,Y, and \space Z$ be random variables. Let $A$ be a subset of $U$ such that $p(X \in U)=1$ Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$ Do these two expressions represent the same thing? ...
1
vote
1answer
33 views

Show that entropy $(p1,…,pi,…,pj,…,pm)$, < entropy $(p1,…, (pi+pj)/2 ,…, (pi+pj)/2 ,…,pm)$.

Show that the entropy of the probability distribution, $(p1,...,pi,...,pj,...,pm)$, is less than the entropy of the distribution $(p1,..., (pi+pj)/2 ,..., (pi+pj)/2 ,...,pm)$. I don't understand what ...
0
votes
0answers
18 views

Applications of dissimilarity measures

A dissimilarity measure of two probability measures $p$ and $q$ is defined as a non negative function $D(p,q)$ which is $0$ iff $p=q$ a.s. The KL divergence is an example of such a function. There are ...
2
votes
0answers
138 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
3
votes
0answers
237 views

Intuition for Fisher information metric

In statistical maniolds $S=\{p_\theta\}$,$\theta=(\theta_1,\dots,\theta_n)$, the Riemaanian metric usually defined is the Fisher information metric $$g_{ij}(\partial_i,\partial_j)=\int \partial_i(\log ...
0
votes
1answer
90 views

mutual information problem

Consider the following problem: What is $I(X;Y)$ where $X$ is the outcome of a roll of a fair 6-sided die and $Y$ is whether the outcome of THAT SAME ROLL was even or odd? Intuitively, I thought ...
7
votes
1answer
248 views

Does “50/50 chance of.. . ” convey information?

I distinctly remember the professor in the undergrad introductory systems & control course saying that "when weather forecasters say there's a 50% chance of precipitation, they are conveying no ...
1
vote
0answers
39 views

How do I measure the similarity of two bivariate time series?

Suppose I have two bivariate time series: $$ ts1 = [<a_1, b_1>, <a_2, b_2>, \cdots, <a_N, b_N>] $$ $$ ts2 = [<c_1, d_2>, <c_2, d_2>, \cdots, <c_N, d_N>] $$ Which ...
11
votes
1answer
310 views

metric in the Wasserstein space of gaussian measures

I am reading the paper "Wasserstein Geometry of Gaussian measures" by Asuka Takatsu (section 3 is of interest to me) and I have difficulties understanding how the metric is used. In particular, I am ...
7
votes
1answer
355 views

Empirical distribution vs. the true one: How fast $KL( \hat{P}_n || Q)$ converges to $KL( P || Q)$?

Let $X_1,X_2,\dots$ be i.i.d. samples drawn from a discrete space $\mathcal{X}$ according to probability distribution $P$, and denote the resulting empirical distribution based on n samples by ...
0
votes
1answer
148 views

p-lim inf definition (limit inferior in probability)

For an arbitrary sequence of real-valued random variables $\{Z_n\}_1^\infty$ , we define limit inferior in probability as follow : $$ p-\liminf_{n\to \infty} Z_n \equiv \sup \{ \beta|\lim_{n\to ...
1
vote
0answers
189 views

Mutual Information of Correlated Bivariate Uniform Distribution

We have correlated bivariate uniform distribution, where X and Y have a correlation coefficient $\rho$ and they uniformly distributed in the following rectangle. What is the mutual information of $X$ ...
9
votes
2answers
585 views

In what sense is the Jeffreys prior invariant?

I've been trying to understand the motivation for the use of the Jeffreys prior in Bayesian statistics. Most texts I've read online make some comment to the effect that the Jeffreys prior is ...
0
votes
1answer
121 views

How to match a discrete distribution to a continuous distribution in information theoretic sense?

Let $$ S \sim N(\mu, \sigma^2) $$ be a normally distributed random variable with known $\mu$ and $\sigma^2$. Suppose, we observe $$ X = \begin{cases} T & \text{if $S \ge 0$}, \\ -T & ...
1
vote
1answer
63 views

Types and Typical sequences

Joint types can often be given in terms of the type of x and a stochastic matrix \begin{equation} V:X\rightarrow Y \end{equation}such that $ P_{x,y}(a,b)=P_{x}(a)V(b|a)$ for every $a\in X$ , $b\in Y$. ...
4
votes
3answers
157 views

simulating a fair random process with an unfair one.

Let's say I have a stochastic process that outputs $1$ or $0$ with probability $p$ or $1-p$ respectively, $p\neq 1/2$. Let's assume this is a repeatable iid process. So I can generate $X_1,X_2\dots$ ...
1
vote
1answer
422 views

How to calculate/approximate expectation of function of a binomial random variable?

I am stuck at following problem in my research. Suppose that $M=m$ is a random variable with binomial distribution and parameters $n,p$. The constants $r$ and $\gamma$ are greater than zero. ...
4
votes
1answer
680 views

What is the relationship of $\mathcal{L}_1$ (total variation) distance to hypothesis testing?

Kullback-Leibler divergence (a.k.a. relative entropy) has a nice property in hypothesis testing: given some observed measurement $m\in \mathcal{Q}$, and two probability distributions $P_0$ and $P_1$ ...
9
votes
2answers
625 views

What is the relationship between the Boltzmann distribution and information theory?

I'm reading a paper on Boltzmann machines (a type of neural network in Machine Learning), and it mentions that "The Boltzmann distribution has some beautiful mathematical properties and it is ...
1
vote
3answers
694 views

Is it wrong to use Binary Vector data in Cosine Similarity?

I am doing Information Retrieval using Cosine Similarity. My data is binary vector. Since most of all reference I read is using non-binary vector (non-binary matrix) data, I am wondering if it is ...
0
votes
1answer
225 views

Relative Entropy given two non-equivalent sets

I am trying to calculate the relative entropy given two collections and have a question regarding some issues. Supposed we have two sets, $Real$ and $Calculated$, and their respective probability ...