Tag Info

New answers tagged


Given: $X$ has Geometric PMF with probability $p$ $H(X) = E(I(X)) = -Elog(P(X)) = -[E(log(p))+{log(1-p)}E(x-1)] = -[log(p)+\frac{1-p}{p}log(1-p)]$ Therefore, $H(X) = -\dfrac{plog(p)+(1-p)log(1-p)}{p}$ where $E(X)= \dfrac{1}{p}$


Let $ P_E = Pr( S \ne \hat{S})$. Then Fano says $$H(S|\hat{S}) \leq h(P_E) + P_E \log_2(|\mathcal{X}| - 1)$$ But $P_E \le \sum P(S_i \ne \hat{S_i})= K P_e$ . Furthermore: $|\mathcal{X}|\le 2^K$ Then (assuming we are in the region $K P_e < 1/2$, where $h(\dot)$ is increasing) we get $$H(S|\hat{S}) \leq h(K P_e) + K P_e \log_2(|\mathcal{X}| - 1) \leq ...


Shannon Entropy, given a discrete probability distribution with probabilities $p_1,p_2,\dots,p_n$ is $$-\sum_i p_i\log_2(p_i).$$ This is not equal to the quantity you gave. If $n=1$ then we must have $p_1=1$ and then the Shannon Entropy is 0. A relevant example is to figure out what $p_1$ maximizes Shannon entropy when $n=2$. This can be thought of as what ...


David answer is right. Regarding your second (wrong) alternative, related to "typical" sequences: the point is not that $n$ is finite. Imagine you have observed 7 coins, with 4 heads and 3 tails. If the next is tail, you would have the same number of tails and heads (the sample average would coincide with the true mean); true, we know (law of large numbers) ...


(1) is correct. A coin has no memory and does not know that it has already come up heads (or whatever) a lot. Your final question is a case of statistical hypothesis testing. If you search Google for "hypothesis testing fair coin" you will get lots of hits.


Just do the math. Let me change notation, call $a_x = P_0[x] $, $b_x=P_1[x] $ and let $e_x=a_x-b_x$ Then each term of the sum (using natural logarithms) has the form $$ a_x \, \ln \frac{a_x}{b_x}=b_x \, \left(1+\frac{e_x}{b_x}\right) \ln \left(1 +\frac{e_x}{b_x}\right) =b_x \, (1+u_x) \, \ln(1+u_x)$$ where $u_x=e_x/b_x$. Now, for small $u_x$, we use ...


Both cases are possible. H(X) [=========================] H(Y) [===============] (original) H(X,Y) [==================================] H(X) [=========================] constant H(Y) [=======] less than original H(X,Y) [=============================] decreases ...

Top 50 recent answers are included