Tag Info

2

$$bb(x)=log_2(x)$$ is the "binary base" , the logarithm to base $2$.

2

The capacity of a continuous-time AWGN channel with bandwith $B$ is (ref), $$C=B \log\left(1+\frac{S}{N_0 B}\right)$$ You just need to let $B\to \infty$ and compute the limit (note that there is an indeterminancy here to resolve, as the first factor tends to $\infty$ and the second to $0$).

1

Information geometry is introduced in Amari's book. A concise treatment of the relation between information theory and statistics can be found in "Information Theory and Statistics : A Tutorial by Csiszar and Shields. Particularly section 3 and 4 are dedicated to information geometry. They are credible researchers on this topic. There is this chapter of ...

1

Without going into too much detail, an extremum w.r.t. $p$ happens as a solution to the Euler-Lagrange equation, which for this functional happens when $p=\frac{1}{e}f$. EDIT: A "little" more detail: in the first case, the Lagrangian is given by $L=L(x,p,p')=p(x)\log\frac{f(x)}{p(x)}=p\left(\log f-\log p\right)$, and since $L$ doesn't depend explicitly on ...

1

The lower bound is correct. Looking for the average word length is equivalent to assuming that all code words occur with equal probability. Then, a lower bound on the expected word length (which equals the average word length now) is the entropy of the code. Since all code words are equally probably, you get $$\log m \le \frac{1}{m}\sum_{i=1}^m s_i.$$ ...

1

Take your geometric random variable distributed as $p_k=\mathbb P (X=k)=p(1-p)^{k}$ for $k\in\{0,1,2,\dots\}$. Note that your entropy is: $$H(X)=-\log p-\frac{1-p}{p}\log(1-p).$$ And consider another random variable $Y$ distributed as $q_k=\mathbb P (Y=k)$ such that: $$\frac{1-p}{p}=\mathbb E(X)=\mathbb E(Y)=\sum_{i=0}^\infty i q_i.$$ We use following ...

1

I do have information after a permutation: I know exactly what the distribution of characters (bytes / alphabet members..) of the plain text was. If I see "trapa" and "olleh", I can certainly tell which one came from "apart".... So it's pretty trivial to win the distinguisher game here (so no perfect secrecy). Added: Another, more formal, way to see ...

1

$\log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $\log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$. There is indeed a ...

1

Note that $$P(X^n\in C_n(t))=P(p(X^n)\ge 2^{-nt})=P\left(-\frac{1}{n}\sum_{1}^n \lg p(X_i)\le t\right)$$ Now, it follows from WLLN, $$\lim_{n\to \infty}P\left(-\frac{1}{n}\sum_{1}^n \lg p(X_i)\le H(X)+\epsilon\right)=1\ \forall \epsilon>0$$ Thus, to have $\lim_{n\to \infty}P[X^n\in C_n(t)]\to 1$, we must have $t\ge \epsilon+H(X)\ \forall ... 1 My educated guess is that you cannot calculate the points on such plots - they are results of extensive simulations. So you need to: Implement a decoding algorithm. In the case of LDPC-codes you need to implement the belief-propagation algorithm. Select a modulation scheme (BPSK, N-QAM, whatever). You also need to know how to calculate the LLRs of ... 1 its says that when the noise standard deviation is .8, the value for Eb/N0 is 1.94. How is this correspondence made? Well, assuming a normalized signal ($x=\pm 1$) [*] and calling$z$the noise, we have $$\frac{\sigma_x^2}{\sigma_z^2}=\frac{1}{(0.8)^2}=1.5625$$ Which in db is$10 \log_{10} 1.5625 =1.9382$dB Update: [*] Why assume$x=\pm 1\$? Just ...

Only top voted, non community-wiki answers of a minimum length are eligible