Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I've always used Shannon's entropy for measuring uncertainty, but I wonder why to use a logarithmic approach. Why shouldn't uncertainty be linear?

For instance, consider the following pairs of distributions:

$$ \left[0.9, 0.1\right], \left[0.99, 0.01\right] $$

$$ \left[0.4, 0.6\right], \left[0.49, 0.51\right] $$

Then you have the following uncertainty measures: $$ H([0.9, 0.1]) = 0.46899559358928122\\ H([0.99, 0.01]) = 0.080793135895911181\\ H([0.4,0.6]) = 0.97095059445466858\\ H([0.49, 0.51]) = 0.9997114417528099 $$

With the following example I just want show that it doesn't satisfy linearity:

$$ H([0.9, 0.1]) - H([0.99, 0.01]) \simeq 0.3882 $$

$$ H([0.49, 0.51]) - H([0.4, 0.6]) \simeq 0.02876 $$

As we can see, distributions that are closer to $[1,0]$ or $[0,1]$ tend faster to zero.

May be this is more a philosophic question but I think that may be someone could give me alternative measures of uncertainty that may be linear or, at least, provide some explanation to the rationale of this approach.


EDIT I don't mean linearity in the whole space but in the intervals $[0,\frac{1}{2}]$ and $[\frac{1}{2}, 1]$. Since, as @r.e.s. comments, is a required property for such a measure that $f(0)<f(\frac{1}{2})$ and $f(1)<f(\frac{1}{2})$

share|improve this question
Why do you expect $ H(X) = -\sum_{i=1}^n {p(x_i) \log_b p(x_i)} $ to be linear? –  draks ... Aug 2 '12 at 10:54
I don't understand how the difference between the entropy of two different distributions measures uncertainty. Uncertainty of what? Is uncertainty a well-defined technical term in information theory that I am unaware of? –  Rahul Aug 2 '12 at 11:08
@draks: I know that $H(X)$ is not linear... I wonder if there is another approach to measure uncertainty that is actually linear. –  synack Aug 2 '12 at 11:49
@RahulNarain: No, the difference doesn't measure the uncertainty, it was just an example of the non-linearity. Yes, in information theory the uncertainty of a discrete random variable is defined as the entropy of its distribution. More info: Entropy –  synack Aug 2 '12 at 11:51
I think you want to measure a different aspect. For you, [0.4,0.4,0.2,0] should have lesser uncertainty than [0.3, 0.3, 0.3, 0.1], right? i.e., no. of maximal entries matters, right? –  Ashok Aug 2 '12 at 14:34

1 Answer 1

A linear function would not have some essential properties reasonably required of such a measure of uncertainty. Consider, as in your examples, distributions $[p, 1-p]$, with $0 \le p \le 1$, on a two-point sample space, and let $h(p)$ be the measure of uncertainty embodied by such a distribution. Among the basic requirements of such a measure are that both $h(0) < h(1/2)$ and $h(1) < h(1/2)$ -- because a probability $p$ equal to $0$ or $1$ corresponds to less uncertainty than does any other value of $p$ -- but this is impossible if $h(p)$ is linear in $p$, or indeed if $h(p)$ is monotonic in $p$. Standard axiomatic developments lead to the concave function $h(p) = - p \log p - (1-p) \log (1-p)$.

share|improve this answer
Yes, you're right. It was my mistake, I didn't mean linearity in the whole space but in $[0, \frac{1}{2}]$ and $[\frac{1}{2}, 1]$, in a way that also verifies the property that you just stated. (So, in the first interval it will monotonically increasing and in the second one, monotonically decreasing). –  synack Aug 2 '12 at 13:33
So, you could imagine the graph of the function that I propose similar to $-|x|$. –  synack Aug 2 '12 at 13:44

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.