# Tag Info

1

Given: $X$ has Geometric PMF with probability $p$ $H(X) = E(I(X)) = -Elog(P(X)) = -[E(log(p))+{log(1-p)}E(x-1)] = -[log(p)+\frac{1-p}{p}log(1-p)]$ Therefore, $H(X) = -\dfrac{plog(p)+(1-p)log(1-p)}{p}$ where $E(X)= \dfrac{1}{p}$

0

Let $P_E = Pr( S \ne \hat{S})$. Then Fano says $$H(S|\hat{S}) \leq h(P_E) + P_E \log_2(|\mathcal{X}| - 1)$$ But $P_E \le \sum P(S_i \ne \hat{S_i})= K P_e$ . Furthermore: $|\mathcal{X}|\le 2^K$ Then (assuming we are in the region $K P_e < 1/2$, where $h(\dot)$ is increasing) we get $$H(S|\hat{S}) \leq h(K P_e) + K P_e \log_2(|\mathcal{X}| - 1) \leq ... 1 Shannon Entropy, given a discrete probability distribution with probabilities p_1,p_2,\dots,p_n is$$-\sum_i p_i\log_2(p_i).$$This is not equal to the quantity you gave. If n=1 then we must have p_1=1 and then the Shannon Entropy is 0. A relevant example is to figure out what p_1 maximizes Shannon entropy when n=2. This can be thought of as what ... 0 David answer is right. Regarding your second (wrong) alternative, related to "typical" sequences: the point is not that n is finite. Imagine you have observed 7 coins, with 4 heads and 3 tails. If the next is tail, you would have the same number of tails and heads (the sample average would coincide with the true mean); true, we know (law of large numbers) ... 4 (1) is correct. A coin has no memory and does not know that it has already come up heads (or whatever) a lot. Your final question is a case of statistical hypothesis testing. If you search Google for "hypothesis testing fair coin" you will get lots of hits. 1 Just do the math. Let me change notation, call a_x = P_0[x] , b_x=P_1[x]  and let e_x=a_x-b_x Then each term of the sum (using natural logarithms) has the form$$ a_x \, \ln \frac{a_x}{b_x}=b_x \, \left(1+\frac{e_x}{b_x}\right) \ln \left(1 +\frac{e_x}{b_x}\right) =b_x \, (1+u_x) \, \ln(1+u_x) where $u_x=e_x/b_x$. Now, for small $u_x$, we use ...

2

Both cases are possible. H(X) [=========================] H(Y) [===============] (original) H(X,Y) [==================================] H(X) [=========================] constant H(Y) [=======] less than original H(X,Y) [=============================] decreases ...

Top 50 recent answers are included