# Tagged Questions

16 views

### Confusion about non-negative mutual information

The formula I was given for calculating information for a specific stimulus $s_x$ is: $$I(R,s_x) = \sum_i p(r_i|s_x) \log_2{p(r_i|s_x)\over p(r_i)}$$ It was also said that information is always ...
139 views

### Similarity between two probability distribution

I am not sure how to put the question. I am not even sure if this question makes sense at all. I know that the similarity of two discrete (or continuous) distributions can be quantified by ...
34 views

### What is the “true” entropy of a binary string?

Consider an infinite binary string $\sigma$ and define its entropy $$H_1 = -(p_0 \log_2 p_0 + p_1 \log_2 p_1)$$ with $p_i = \lim_{N\rightarrow \infty} N(i)/N$, $N(i)$ the number of $i$'s among the ...
55 views

### Joint Probability from Marginal Probabilities

$X, Y_1, Y_2$ are random variables with (possibly) different finite alphabets. For given conditional probability mass functions $\mathbb{P}(Y_1|X)$ and $\mathbb{P}(Y_2|X)$, is it always possible to ...
48 views

76 views

### About the differential entropies of well-known continuous distributions

Assume that the continuous random variable $X$ has a distribution (in a closed form expression) with differential entropy $h(X)$. Q) Then, is it true for any continuous distribution that the ...
577 views

### Binary symmetric channel capacity or mutual information inequality

I proved that I(X,Y) <= 1 - H(p) to the following way: How can I prove if I start in that way I(X,Y) = H(X) - H(X|Y), I ...
178 views

### Simple trace distance problem

I am self studying a course on information theory and came with the following question: $A$ and $B$ represent two possibly different probability distributions representing two different independent ...
155 views

### rényi entropy as a derivative

Let $x=(x_i)$ be a probability measure on $\{1,\ldots,n\}$. Suppose $1<p<\infty$. The Rényi entropy of $x$ is $$H^p(x)=\frac{1}{1-p}\log \sum_{i} x_i^p.$$ Does there exist a formula for ...
86 views

### Variational distance basic properties

The variational distance between two probability distributions $X$ and $Y$ taking values on the same alphabet $\mathcal A$ is defined as \delta (X,Y)=1/2\sum_{a\in A} ...
252 views

### Explicit examples of smooth entropy computation

Smooth classic entropies generalize the standard notions of entropy. This smoothing stands for a minimization/maximization over all events $\Omega$ such that $p(\Omega)\geq 1-\varepsilon$ for a given ...
Let $X$ be a discrete random variable with Laplacian distribution with mean $0$ and scale $\lambda$, as  p(X) = \frac{1}{2\lambda} \exp\left(-\frac{|x|}{2\lambda}\right), \\ X \in ...