Tagged Questions
0
votes
0answers
38 views
Intregral of exponential of Shannon Entropy Function
Here I am going to ask a similar question as rde asked , that is what is the integral of exponential of entropy function. That is what is the value of
$F[H(x)]=\int_{-1}^{+1} e^{ikH(f(x^2))} dx$
...
1
vote
0answers
29 views
convergence of discrete random variables with finite entropy
Let $Z$ be the set of discrete random variables on some probability space. Define the quantity $d(X_1,X_2)=h(X_1 \mid X_2)+h(X_2 \mid X_1)$ between two random variables $X_1, X_2 \in Z$. For $X \in Z$ ...
0
votes
1answer
186 views
Kullback-Leibler distance between 2 probability distributions
Can I determine the Kullback-Leibler distance
$$
D_{\mathrm{KL}}(P\parallel Q)=\sum_{i}\ln\left(\frac{P(i)}{Q(i)}\right) P(i)
$$
between the following probability distributions?
...
1
vote
1answer
162 views
Explicit examples of smooth entropy computation
Smooth classic entropies generalize the standard notions of entropy. This smoothing stands for a minimization/maximization over all events $\Omega$ such that $p(\Omega)\geq 1-\varepsilon$ for a given ...
1
vote
0answers
87 views
Incrementally compute the conditional entropy
Is it possible to compute a conditional entropy (see the two following formulas) in an incremental manner ? That is, the sets C and K are not fix: each time we have a new element c, set K may increase ...
1
vote
1answer
72 views
Question about Mutual Information
I am learning about mutual information, and am confused about one of the definitions. Mutual information is defined as
$ I(X;Y) = H(X) - H(X | Y) $
where,
$$ H(X) = \sum_{x} p(x) \log ...
3
votes
2answers
441 views
Can I normalize KL-divergence to be $\leq 1$?
The Kullback-Leibler divergence has a strong relationship with mutual information, and mutual information has a number of normalized variants. Is there some similar, entropy-like value that I can use ...
