1
vote
0answers
7 views

Gaussian distribution variance estimation

It's well known if I have a process generating normally distribuited data, I can estimate the parameters of the gaussian function: ...
1
vote
0answers
31 views

What is the “true” entropy of a binary string?

Consider an infinite binary string $\sigma$ and define its entropy $$H_1 = -(p_0 \log_2 p_0 + p_1 \log_2 p_1)$$ with $p_i = \lim_{N\rightarrow \infty} N(i)/N$, $N(i)$ the number of $i$'s among the ...
1
vote
2answers
44 views

Entropy of a distribution over strings

Suppose for some parameter $d$, we choose a string from the Hamming cube ($\{0,1\}^d$) by setting each bit to be $0$ with probability $p$ and $1$ with probability $1-p$. What is the entropy of this ...
1
vote
1answer
51 views

Kullback–Leibler divergence in bits

Well known formula of KL divergence when we have a discrete probability distributions. $$D_{KL}(P \parallel Q)=\sum\limits_i \ln \left(\frac{P(i)}{Q(i)}\right) P(i)$$ Can someone explain why the ...
1
vote
0answers
62 views

what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
2
votes
1answer
88 views

Notion of Relative Entropy

I do not understand the notion of relative entropy. Relative Entropy. $D_{KL}(P||Q) = \sum_{i}^{}P(i)\log \frac{P(i)}{Q(i)}$. I try to get some intuition why it looks the way it looks. I see that it ...
0
votes
1answer
125 views

What is the maximum entropy distribution of points on a sphere that has a fixed non-zero average cosine of the polar angle?

Suppose we have a unit vector in 3D space whose orientation has some unknown distribution $p(\theta,\phi)$. All we know about this distribution is the average value of $cos(\theta)$: ...
3
votes
1answer
341 views

How Entropy scales with sample size

For a discrete probability distribution, the entropy is defined as: $$H(p) = \sum_i p(x_i) \log(p(x_i))$$ I'm trying to use the entropy as a measure of how "flat / noisy" vs. "peaked" a distribution ...
1
vote
1answer
51 views

Constructing Distribution By Coin Flipping

I am interested in any example of construction distribution by coin flipping. Actually I want to show the process of construction a random variable $X$ with distribution $(p_1,...,p_n)$ by coin ...
0
votes
0answers
136 views

Replace a continuous probability distribution with a discrete one

Say one wants to fit a curve $f(x)$ to a set of noisy data points $(x_i, y_i)$. If the error for each point $y_i$ is assumed to be normally distributed with variance $\sigma_i^2$, one wants to find ...
1
vote
1answer
75 views

About the differential entropies of well-known continuous distributions

Assume that the continuous random variable $X$ has a distribution (in a closed form expression) with differential entropy $h(X)$. Q) Then, is it true for any continuous distribution that the ...
2
votes
1answer
62 views

What is the CONCEPT when we speak of maximum entropy?

What is an intuitive interpretation of the concept of maximum entropy? I want to understand this concept better but what I'm finding is too "advanced" right now. Can anyone simplify it ... imagine I'm ...
0
votes
1answer
580 views

Kullback-Leibler distance between 2 probability distributions

Can I determine the Kullback-Leibler distance $$ D_{\mathrm{KL}}(P\parallel Q)=\sum_{i}\ln\left(\frac{P(i)}{Q(i)}\right) P(i) $$ between the following probability distributions? ...
1
vote
1answer
148 views

Entropy of Zipf and Zeta Distributions

I was wondering how to show entropy of the zeta distribution. It is: $$ H_\mathrm{zeta}(X) = \sum_{k=1}^\infty \frac{1/k^s}{\zeta(s)} \log(k^s \zeta(s))$$ The entropy of the zipf distribution is: ...
1
vote
1answer
89 views

Entropy of Order Statistic

Consider $n$ independent and identically distributed random variables $ \{X_i\}_{i=1,...n} $ with support on some interval $[a,b]$ and its $n$'th order statistic $\max_{i \in \{1,...n\}} X_i$ . The ...
2
votes
0answers
150 views

rényi entropy as a derivative

Let $x=(x_i)$ be a probability measure on $\{1,\ldots,n\}$. Suppose $1<p<\infty$. The Rényi entropy of $x$ is $$ H^p(x)=\frac{1}{1-p}\log \sum_{i} x_i^p. $$ Does there exist a formula for ...
1
vote
1answer
246 views

Explicit examples of smooth entropy computation

Smooth classic entropies generalize the standard notions of entropy. This smoothing stands for a minimization/maximization over all events $\Omega$ such that $p(\Omega)\geq 1-\varepsilon$ for a given ...
2
votes
1answer
211 views

Parameter estimation for a distribution by minimizing its conditional entropy

Let $X$ be a discrete random variable with Laplacian distribution with mean $0$ and scale $\lambda$, as $$ p(X) = \frac{1}{2\lambda} \exp\left(-\frac{|x|}{2\lambda}\right), \\ X \in ...
1
vote
0answers
79 views

Showing that Normalized Redundancy is nonreliant on the properties of Bijection and Monotonicity

In information theory, the concept of mutual information states that for two features of arbitrary discretized probability, the following formula holds true: \begin{aligned} I(X;Y) = \sum_{y \in Y} ...
4
votes
2answers
996 views

Derivation of the maximum entropy distribution

I am reading a book and having trouble following something. The problem is to try to maximize the differential entropy $-\int_{0}^{\infty}p(r)\log p(r)$ under the constraints that ...