Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Consider a binary message in which $0$ has has probability $1/3$ and $1$ has probability $2/3$. What value of $H$ should be assign?

I know that you split up $1$ into two messages $1a$ and $1b$. Then I think you have to use conditional probability here. Am I on the right track? So $$\left[\text{entropy of} \ (0,1a, 1b) \ \text{message} \right] = \left[ \text{entropy of} \ (0,1) \ \text{message} \right] + \text{something}$$

share|cite|improve this question
What does "split up 1 into two messages 1a and 1b" mean? – Emre May 1 '11 at 4:14
I agree with Emre. There is no reason to split 1 into two messages 1a and 1b (unless you are looking at a problem which explicitly involves such a split, in which case you should specify it properly in your question.) – svenkatr May 1 '11 at 6:12

Could you define $H$ so others don't have to look? Wikipedia says $H(x)=-\sum p(x_i)\log(p(x_i)).$ You have the $p(x_i)$ so what more do you need? Entropy is additive.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.