0
votes
1answer
30 views

Probability density function of entropy of a gaussian variable

I have a problem finding the probability density function of entropy of a normally distributed sample. It is known that the entropy of a gaussian variable $X$ equals $H=h(X)={1\over2}\log(2\pi ...
4
votes
2answers
42 views

Determinant of Fisher information

In information geometry, the determinant of the Fisher information matrix is a natural volume form on a statistical manifold, so it has a nice geometrical interpretation. But what is it in ...
-1
votes
0answers
26 views

Hypercontractivity of Markov Operator

I have been reading a paper by Ahlswede and Gacs on hypercontractivity of Markov operator (see here 1) and its application to information theory. To be honest, I could not fully understand the ...
0
votes
0answers
49 views

Special case of Kullback-Leibler additivity

I have three random variables $X,Y,Z$. If $(X,Z)$ are an independent pair and $(Y,Z)$ are an independent pair, then the additive property of the Kullback-Leibler divergence says $K(X,Z|Y,Z) = K(X|Y) ...
3
votes
1answer
54 views

Upper Bound on Mutual Information

I am interested in an upper bound on mutual information that I have been encountering frequently in the statistics and probability literature. I have yet to see the "purest" form of the inequality, so ...
0
votes
1answer
34 views

A question on Markov chain

Suppose for two random variables $X$ and $Y$ we have $X\perp\!\!\!\perp Y$ and also assume that three random variables $X$, $Y$ and $Z$ form the following Markov chain: $X\to Z\to Y$. Do these two ...
0
votes
1answer
19 views

Loss of information while projecting multidimensional data

I'm interested in the evaluation of the loss of information after projecting multidimensional data. Since the dimensional reduction is a common tool to analyse data,a question about the loss of ...
1
vote
1answer
123 views

The golden ratio in statistics of literature

Let a book, for example, or a poem... It consists in words and letters and symbols like : ;,!... Let $W_b$=the number of words of the book. Let $L_b$=the number of letters of the book. The number ...
4
votes
1answer
58 views

Joint Probability from Marginal Probabilities

$X, Y_1, Y_2$ are random variables with (possibly) different finite alphabets. For given conditional probability mass functions $\mathbb{P}(Y_1|X)$ and $\mathbb{P}(Y_2|X)$, is it always possible to ...
2
votes
3answers
281 views

Is Standard Deviation the same as Entropy?

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution ...
2
votes
1answer
38 views

Is it true that $H(X|Y)=H(Y|X)$?

I have some difficulties with the question whether $H(X|Y)=H(Y|X)$? From my knowledge $I(X;Y)=H(X)-H(X|Y) = H(Y)-H(Y|X)$ so $H(X|Y)=H(Y|X)$ only when $H(X)=H(Y)$ The question is whether it's ...
3
votes
3answers
145 views

Estimating the entropy

Given a discrete random variable $X$, I would like to estimate the entropy of $Y=f(X)$ by sampling. I can sample uniformly from $X$. The samples are just random vectors of length $n$ where the entries ...
0
votes
2answers
50 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
2
votes
1answer
64 views

Confused about notation: difference between $\prod_{i=1}^np(x_i)$ and $\prod_{i=1}^np(x)$

In my information theory book by Cover and Thomas, at the beginning of the channel coding theorem, it's written: "Each entry in this matrix" (the matrix of the randomly generated code) "is ...
0
votes
1answer
161 views

If $f_\theta=Uniform(\theta,\theta +1)$, a sufficient statistic for $\theta$ is… but why?

If $f_\theta=\mathrm{Uniform}(\theta,\theta +1)$, a sufficient statistic for $\theta$ is $$T(X_1,X_2,\dots,X_n)=(\max\lbrace X_1,X_2,\dots,X_n\rbrace,\min\lbrace X_1,X_2,\dots,X_n\rbrace).$$ Can ...
3
votes
2answers
175 views

Markov chains for beginners, how to think about them?

So this is what my book states: Random variables $X,Y, and Z$ are said to form a Markov chain in that order denoted $X\rightarrow Y \rightarrow Z$ if and only if: $p(x,y,z)=p(x)p(y|x)p(z|y) $ ...
2
votes
2answers
272 views

Random process, stochastic process explained intuitively?

So I've read the definitions online and this is what I understood. $X(t)$ is a random process for $t>0$ and we can think of it as being a random variable at any given time $t=t_0$. For example, ...
1
vote
2answers
286 views

“by definition A and B R.V are independent means that: $p(A∪B)=p(A)+p(B)$ right?” No, absolutely not right.

Can someone please explain why? Isn't $p(a,b)=p(a)*p(b) $ equivalent to $p(A∪B)=p(A)+p(B)$? If not can you please give a counterexample or something? Thanks a lot!
2
votes
1answer
89 views

I.I.D what does this stand for?

So almost everywhere in the book it's written "random variables are IID", what does this mean? I think it means independent and identically distributed but not sure. So by definition A and B R.V are ...
1
vote
1answer
37 views

Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$

Let $X,Y, and \space Z$ be random variables. Let $A$ be a subset of $U$ such that $p(X \in U)=1$ Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$ Do these two expressions represent the same thing? ...
1
vote
1answer
33 views

Show that entropy $(p1,…,pi,…,pj,…,pm)$, < entropy $(p1,…, (pi+pj)/2 ,…, (pi+pj)/2 ,…,pm)$.

Show that the entropy of the probability distribution, $(p1,...,pi,...,pj,...,pm)$, is less than the entropy of the distribution $(p1,..., (pi+pj)/2 ,..., (pi+pj)/2 ,...,pm)$. I don't understand what ...
3
votes
0answers
147 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
3
votes
0answers
246 views

Intuition for Fisher information metric

In statistical maniolds $S=\{p_\theta\}$,$\theta=(\theta_1,\dots,\theta_n)$, the Riemaanian metric usually defined is the Fisher information metric $$g_{ij}(\partial_i,\partial_j)=\int \partial_i(\log ...
0
votes
1answer
94 views

mutual information problem

Consider the following problem: What is $I(X;Y)$ where $X$ is the outcome of a roll of a fair 6-sided die and $Y$ is whether the outcome of THAT SAME ROLL was even or odd? Intuitively, I thought ...
7
votes
1answer
254 views

Does “50/50 chance of.. . ” convey information?

I distinctly remember the professor in the undergrad introductory systems & control course saying that "when weather forecasters say there's a 50% chance of precipitation, they are conveying no ...
1
vote
0answers
40 views

How do I measure the similarity of two bivariate time series?

Suppose I have two bivariate time series: $$ ts1 = [<a_1, b_1>, <a_2, b_2>, \cdots, <a_N, b_N>] $$ $$ ts2 = [<c_1, d_2>, <c_2, d_2>, \cdots, <c_N, d_N>] $$ Which ...
11
votes
1answer
328 views

metric in the Wasserstein space of gaussian measures

I am reading the paper "Wasserstein Geometry of Gaussian measures" by Asuka Takatsu (section 3 is of interest to me) and I have difficulties understanding how the metric is used. In particular, I am ...
7
votes
1answer
365 views

Empirical distribution vs. the true one: How fast $KL( \hat{P}_n || Q)$ converges to $KL( P || Q)$?

Let $X_1,X_2,\dots$ be i.i.d. samples drawn from a discrete space $\mathcal{X}$ according to probability distribution $P$, and denote the resulting empirical distribution based on n samples by ...
0
votes
1answer
154 views

p-lim inf definition (limit inferior in probability)

For an arbitrary sequence of real-valued random variables $\{Z_n\}_1^\infty$ , we define limit inferior in probability as follow : $$ p-\liminf_{n\to \infty} Z_n \equiv \sup \{ \beta|\lim_{n\to ...
1
vote
0answers
192 views

Mutual Information of Correlated Bivariate Uniform Distribution

We have correlated bivariate uniform distribution, where X and Y have a correlation coefficient $\rho$ and they uniformly distributed in the following rectangle. What is the mutual information of $X$ ...
9
votes
2answers
617 views

In what sense is the Jeffreys prior invariant?

I've been trying to understand the motivation for the use of the Jeffreys prior in Bayesian statistics. Most texts I've read online make some comment to the effect that the Jeffreys prior is ...
0
votes
1answer
124 views

How to match a discrete distribution to a continuous distribution in information theoretic sense?

Let $$ S \sim N(\mu, \sigma^2) $$ be a normally distributed random variable with known $\mu$ and $\sigma^2$. Suppose, we observe $$ X = \begin{cases} T & \text{if $S \ge 0$}, \\ -T & ...
1
vote
1answer
64 views

Types and Typical sequences

Joint types can often be given in terms of the type of x and a stochastic matrix \begin{equation} V:X\rightarrow Y \end{equation}such that $ P_{x,y}(a,b)=P_{x}(a)V(b|a)$ for every $a\in X$ , $b\in Y$. ...
4
votes
3answers
157 views

simulating a fair random process with an unfair one.

Let's say I have a stochastic process that outputs $1$ or $0$ with probability $p$ or $1-p$ respectively, $p\neq 1/2$. Let's assume this is a repeatable iid process. So I can generate $X_1,X_2\dots$ ...
1
vote
1answer
428 views

How to calculate/approximate expectation of function of a binomial random variable?

I am stuck at following problem in my research. Suppose that $M=m$ is a random variable with binomial distribution and parameters $n,p$. The constants $r$ and $\gamma$ are greater than zero. ...
4
votes
1answer
690 views

What is the relationship of $\mathcal{L}_1$ (total variation) distance to hypothesis testing?

Kullback-Leibler divergence (a.k.a. relative entropy) has a nice property in hypothesis testing: given some observed measurement $m\in \mathcal{Q}$, and two probability distributions $P_0$ and $P_1$ ...
9
votes
2answers
643 views

What is the relationship between the Boltzmann distribution and information theory?

I'm reading a paper on Boltzmann machines (a type of neural network in Machine Learning), and it mentions that "The Boltzmann distribution has some beautiful mathematical properties and it is ...
1
vote
3answers
730 views

Is it wrong to use Binary Vector data in Cosine Similarity?

I am doing Information Retrieval using Cosine Similarity. My data is binary vector. Since most of all reference I read is using non-binary vector (non-binary matrix) data, I am wondering if it is ...
0
votes
1answer
233 views

Relative Entropy given two non-equivalent sets

I am trying to calculate the relative entropy given two collections and have a question regarding some issues. Supposed we have two sets, $Real$ and $Calculated$, and their respective probability ...