I'm confused by what it means when it says rate. At the bottom of page 15 and page 16 here http://www0.maths.ox.ac.uk/system/files/coursematerial/2014/3093/6/Lecture_notes.pdf it seems to be saying that the rate is the same as the entropy of the source. Shouldn't it instead say that we need $H/(log_2k)$ to be less than the channel capacity, where each source symbol $x\in A$ is coded by $c:\ A\to\ B^k$ as $c(x)$ which has length k?


Entropy $H$ as stated is per source symbol. A source output of $n$ symbols has entropy $nH$. Note that the Theorem on p. 15 states that you're encoding a message of length $n.$

  • $\begingroup$ Shouldn't each output $b \in B^k$ have the same entropy as one symbol in $A$ though (if $c$ is uniquely decipherable)? Why would encoding one symbol as a word of length k increase the entropy? $\endgroup$
    – user152440
    May 14 '15 at 12:32
  • $\begingroup$ See edited answer $\endgroup$
    – kodlu
    May 14 '15 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.