I have a small question I came by , but I am not quite sure about the right approach to it.
Suppose that we have a source that transmits 5 symbols. We have two cases.
- When all symbols got the same chance.
- When one symbol got 1/2 chance and the rest got the same chance.
Question: In which of the above two case does the source coding works better and why?
Intuitively, I feel that in second case, source coding works better.
My attempt to justify this, is through Shannon's entropy! (though I am not sure if I am correct!)
Case1 entropy would be: H1= log2(5) = 2.32 bits/symbol
Case2 entropy: H2= 1/2log2(2) + 1/8log2(8)*4 = 2bits/symbol.
In case 2 , we know more info about the source and therefore we need less bit to represent it.
Any opinions on this? Cheers!