0
votes
3answers
75 views

Similarity between two probability distribution

I am not sure how to put the question. I am not even sure if this question makes sense at all. I know that the similarity of two discrete (or continuous) distributions can be quantified by ...
1
vote
0answers
31 views

What is the “true” entropy of a binary string?

Consider an infinite binary string $\sigma$ and define its entropy $$H_1 = -(p_0 \log_2 p_0 + p_1 \log_2 p_1)$$ with $p_i = \lim_{N\rightarrow \infty} N(i)/N$, $N(i)$ the number of $i$'s among the ...
4
votes
1answer
50 views

Joint Probability from Marginal Probabilities

$X, Y_1, Y_2$ are random variables with (possibly) different finite alphabets. For given conditional probability mass functions $\mathbb{P}(Y_1|X)$ and $\mathbb{P}(Y_2|X)$, is it always possible to ...
1
vote
1answer
44 views

locally linearize a CDF

I have a sequence of discrete CDF's that converge to continuous CDF. Assume we call it $F_n(x)$. If say at some point, say $R$, $F_n$ is differentiable, then we can write $F_n(R+\xi) \approx ...
1
vote
0answers
57 views

Negative exponential/ exponential power distribution between 0 .0 and 1.0?

Note: I'm not very familiar with distribution and higher level math Heyho, I'm currently looking for a way to generate random values between 0.0 and 1.0 with an exponential power or negative ...
1
vote
0answers
24 views

Does this quantity have a meaning or relation to something else?

$X$ is a discrete random variable taking values in $\{0,1,2,3,\ldots\}$ with a probability mass function $p_X(n)$. Let $$U_k(X)=\sum_{n=k}^\infty\sqrt{np_X(n)p_X(n-k)}$$ where $k$ is a positive ...
1
vote
0answers
62 views

what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
0
votes
1answer
72 views

A quick chanllenge: height and weight probability problem

The average height and weight of a group of people is 175cm and 70kg; Find the upper bound of the portion of the people who are over 200cm and over 100kg. I thought about Markov inequality, but I ...
2
votes
1answer
89 views

Notion of Relative Entropy

I do not understand the notion of relative entropy. Relative Entropy. $D_{KL}(P||Q) = \sum_{i}^{}P(i)\log \frac{P(i)}{Q(i)}$. I try to get some intuition why it looks the way it looks. I see that it ...
0
votes
0answers
62 views

Average min-entropy and statistical distance

Let $X$ be a random variable over a set $\mathcal{X}$. The min-entropy of $X$ is defined as $$ H_{\infty}(X) := -\log(\max_x \mathbb{P}_{x\leftarrow X}[X=x]). $$ For a pair of (possibly correlated) ...
1
vote
0answers
124 views

KL divergence of multinomial distribution

Consider $q(x)$ be a Multinomial distribution over $\{1, \ldots, k\}$ with parameters $\{\theta_1,\ldots, \theta_k\}$. And p(x) over $\{1,\ldots, k\}$ with distribution $p(x)=\frac{1}{k}$. Then what ...
2
votes
2answers
104 views

Understanding notation difference between mutual information and information divergance

The mutual information is defined on random variables. That is, $I(X;Y)$ denotes the mutual information between random variables $X$ and $Y$. On the other hand, the the Kullback-Leibler divergence is ...
2
votes
0answers
120 views

Joint distribution between a uniform random variable and a function which is “almost” independent from it

Motivation Let $f(\cdot)$ be a (possibly randomized) function, such that for any random variable $X$ (taking values from a finite set $D$), $X$ and $f(X)$ are statistically independent. Let $U, U_1, ...
1
vote
1answer
75 views

About the differential entropies of well-known continuous distributions

Assume that the continuous random variable $X$ has a distribution (in a closed form expression) with differential entropy $h(X)$. Q) Then, is it true for any continuous distribution that the ...
0
votes
0answers
551 views

Binary symmetric channel capacity or mutual information inequality

I proved that I(X,Y) <= 1 - H(p) to the following way: How can I prove if I start in that way I(X,Y) = H(X) - H(X|Y), I ...
1
vote
1answer
175 views

Simple trace distance problem

I am self studying a course on information theory and came with the following question: $A$ and $B$ represent two possibly different probability distributions representing two different independent ...
2
votes
0answers
150 views

rényi entropy as a derivative

Let $x=(x_i)$ be a probability measure on $\{1,\ldots,n\}$. Suppose $1<p<\infty$. The Rényi entropy of $x$ is $$ H^p(x)=\frac{1}{1-p}\log \sum_{i} x_i^p. $$ Does there exist a formula for ...
1
vote
1answer
85 views

Variational distance basic properties

The variational distance between two probability distributions $X$ and $Y$ taking values on the same alphabet $\mathcal A$ is defined as \begin{equation} \delta (X,Y)=1/2\sum_{a\in A} ...
1
vote
1answer
246 views

Explicit examples of smooth entropy computation

Smooth classic entropies generalize the standard notions of entropy. This smoothing stands for a minimization/maximization over all events $\Omega$ such that $p(\Omega)\geq 1-\varepsilon$ for a given ...
2
votes
1answer
211 views

Parameter estimation for a distribution by minimizing its conditional entropy

Let $X$ be a discrete random variable with Laplacian distribution with mean $0$ and scale $\lambda$, as $$ p(X) = \frac{1}{2\lambda} \exp\left(-\frac{|x|}{2\lambda}\right), \\ X \in ...
1
vote
0answers
79 views

Showing that Normalized Redundancy is nonreliant on the properties of Bijection and Monotonicity

In information theory, the concept of mutual information states that for two features of arbitrary discretized probability, the following formula holds true: \begin{aligned} I(X;Y) = \sum_{y \in Y} ...