A communication company uses a system to transmit four different symbols ${S_1, S_2, S_3, S_4}$. Each symbol has a probability to occur according to the following table \begin{equation*} \begin{array}{c|c|c|c|c} & S_1 & S_2 & S_3 & S_4 \\ \hline p_i & 0.05 & 0.61 & 0.27 & 0.07 \end{array} \end{equation*} Calculate the entropy of the system and the minimum number of bits required for transmission.

So I already calculated the entropy. Regarding the minimum number of bits required for transmission at first I thought "this is straight forward" and calculated

\begin{equation} \lceil \log_2 4 \rceil = \lceil 2 \rceil = 2 \end{equation}

as we have four symbols but now I am not so sure anymore because I didnt take the probalities into account at all. I am afraid I misunderstood the minimum number of bits required to represent four symbols as the the thing I am actually asked to calculate here.

I am sorry in advance if this is super trivial. I just started with information theory and so on and am very new to this.

So how do I calculate the minimum number of bits required for transmission?

  • 1
    $\begingroup$ By definition, entropy is theoretically the minimum number of bits that are required for transmission. However, it may not be possible to achieve that in practice. You can generate a Huffman code for the same and compute the expected number of bits required for transmission. Huffman code is the optimal coding scheme for a finite probability distribution. $\endgroup$ – sudeep5221 May 25 at 19:40
  • $\begingroup$ Many thanks for this hint. The professor didnt even mention "Huffman code"...So I will look into this...I have to admit that I dont really understand what entropy is in information theory. $\endgroup$ – u49K3df2 May 25 at 19:45
  • 1
    $\begingroup$ No problem. :) Entropy is slightly subtle concept and might take time to have a deeper understanding of it. To loosely define it, it's a measure of 'uncertainty' of a random variable. Consider a binary rv such that $P(X = 1) = 0.5$. Then in this case you are totally uncertain what the value of realisation of such a rv would be. On the other hand if $P(X = 1) = 0.9$, then you're reasonably sure that the value will be $1$. Hence there's lesser uncertainty in the second case. This is what entropy tries to capture. It's a very loose explanation but I hope it helps you to get an intuition. $\endgroup$ – sudeep5221 May 25 at 19:57
  • 1
    $\begingroup$ This actually gave me more insight than anything else I have been reading about entropy so far!! $\endgroup$ – u49K3df2 May 25 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.