# Tag Info

## New answers tagged entropy

3

As far as I can see the problem is that $\partial_{\theta_{i}} E_{\theta}[\ln(p)] \neq E_{\theta}[\partial_{\theta_{i}} \ln(p)]$ You wrote $E_{\theta}[\partial_{\theta_{i}} \ln(p)]$ correctly while the other term is $$\partial_{\theta_{i}} E_{\theta}[\ln(p)] = \int \left ( \partial_{\theta_{i}} \ln(p(x;\theta)) \right ) p(x;\theta) dx + \int ... 2 The error is here:$$ E_{\theta}\left [ \frac{\partial}{\partial \theta_{i}} \ln(p(x; \theta)) \right ] = \frac{\partial}{\partial \theta_{i}} E_{\theta} \left [ \ln(p(x; \theta)) \right ] $$This is incorrect as the density depends on \theta: when writing the derivative there is one additional term$$ E_\theta\left[\ln p(x;\theta) \frac{\partial ...

0

I do not fully understand the expansion in the OP; in absence of further details I would like to consider the limit $q\rightarrow 1$ as follows. Let $x>0$; then $\ln_q(x)=\frac{e^{(1-q)\ln x}-1}{1-q}$ by definition. To consider the limit $q\rightarrow 1$, we expand $\ln_q x$ around $q=1$, i.e. $$\ln_q(x)=\frac{e^{(1-q)\ln ... 2 You've got the wrong support for the events' variables. An even number either comes up or it does not. This event can be measured by an random variable which takes the values 1 when it happens, or 0 if it does not. Likewise for the event of a number divisible by 3. Thus X\in\{0,1\} and Y\in\{0,1\} are indicator random variables, that ... 1 Compare the following two definitions: (i) Rate R is achievable for a given channel if there exists a sequence of \left(2^{nR},n \right)-codes such that the probability of decoding error goes to zero as n goes to infinity. Shannon capacity is the supremum of all achievable rates. (ii) Rate R is achievable with zero error for a given channel if ... 0 This theorem has not a specific name, but a proof for this has been offered in the following article on page 4: http://lthiwww.epfl.ch/~leveque/Projects/telatar.pdf 2 Use the definition. We have$$H(p_1,p_2,\ldots,p_n) = -\sum_{k=1}^np_k\log p_k$$and$$H(p_1+p_2,\ldots,p_n) = -(p_1+p_2)\log(p_1+p_2) - \sum_{k=3}^np_k\log p_k$$so$$H(p_1,p_2,\ldots,p_n) - H(p_1+p_2,\ldots,p_n) = -p_1\log p_1 - p_2\log p_2 + (p_1+p_2)\log (p_1+p_2)$$and the right hand side can be written (after a bit of algebra):$$-(p_1+p_2)\left( ...

1

Intuitively for proof I drew a pie chart, with probabilities $p_1$, $p_2$, and $p_*$ for all other probabilities. Then the Entropy formula is something like: what is the expected number of bits would I need to describe picking a random piece of the pie with given probabilities. That event can also be described as an event of picking either the piece with ...

0

Probably way too late for you, but as a future reference. The $p_n$ you calculate is the fraction and not the probability of a complete random event. Probably there is a lot of correlation in your data. Suppose you'd have a file of 1MB pure 0, followed by 1MB of pure 1 (or 255 to be more accurate for bytes). Your fraction of bits set to true is 50%, and ...

0

Define $$h(p) := -\int_{-\infty}^{\infty}f_1(x,p)\log f_1(x,p)\,dx$$ and show that for some $p\in(0,1)$, $$h(p) > h(1/2).$$ This can be done by showing that $h$ has a maximum at some $p \in(0,1)\backslash\{1/2\}$. Hints: Find $p$ such that $\frac{d}{dp}h(p) = 0$ and $\frac{d^2}{dp^2}h(p) < 0$. ...

1

Use the normalized entropy: $$H_n(p) = -\sum_i \frac{p_i \log_b p_i}{\log_b n}.$$ For a vector $p_i = \frac{1}{n}\ \ \forall \ \ i = 1,...,n$, the Shannon entropy is maximized. Normalizing the entropy by $\log_b n$ gives $H_n(p) \in [0, 1]$. You will see that this is simply a change of base, so one may drop the normalization term and set $b = n$. You can ...

0

What you've calculated in Method 1 is the change in entropy from all five dice showing not 1 to all five dice showing 1.   You haven't included the possibility that one or more die starts out showing 1.   Hence you've counted only $5^5$ microstates. What you've calculated in Method 2 is the change in entropy from some of the five dice show not 1 ...

Top 50 recent answers are included