Linearity of uncertainty

I've always used Shannon's entropy for measuring uncertainty, but I wonder why to use a logarithmic approach. Why shouldn't uncertainty be linear?

For instance, consider the following pairs of distributions:

$$\left[0.9, 0.1\right], \left[0.99, 0.01\right]$$

$$\left[0.4, 0.6\right], \left[0.49, 0.51\right]$$

Then you have the following uncertainty measures: $$H([0.9, 0.1]) = 0.46899559358928122\\ H([0.99, 0.01]) = 0.080793135895911181\\ H([0.4,0.6]) = 0.97095059445466858\\ H([0.49, 0.51]) = 0.9997114417528099$$

With the following example I just want show that it doesn't satisfy linearity:

$$H([0.9, 0.1]) - H([0.99, 0.01]) \simeq 0.3882$$

$$H([0.49, 0.51]) - H([0.4, 0.6]) \simeq 0.02876$$

As we can see, distributions that are closer to $[1,0]$ or $[0,1]$ tend faster to zero.

May be this is more a philosophic question but I think that may be someone could give me alternative measures of uncertainty that may be linear or, at least, provide some explanation to the rationale of this approach.

Thanks!

EDIT I don't mean linearity in the whole space but in the intervals $[0,\frac{1}{2}]$ and $[\frac{1}{2}, 1]$. Since, as @r.e.s. comments, is a required property for such a measure that $f(0)<f(\frac{1}{2})$ and $f(1)<f(\frac{1}{2})$

-
Why do you expect $H(X) = -\sum_{i=1}^n {p(x_i) \log_b p(x_i)}$ to be linear? – draks ... Aug 2 '12 at 10:54
I don't understand how the difference between the entropy of two different distributions measures uncertainty. Uncertainty of what? Is uncertainty a well-defined technical term in information theory that I am unaware of? – Rahul Aug 2 '12 at 11:08
@draks: I know that $H(X)$ is not linear... I wonder if there is another approach to measure uncertainty that is actually linear. – synack Aug 2 '12 at 11:49
@RahulNarain: No, the difference doesn't measure the uncertainty, it was just an example of the non-linearity. Yes, in information theory the uncertainty of a discrete random variable is defined as the entropy of its distribution. More info: Entropy – synack Aug 2 '12 at 11:51
I think you want to measure a different aspect. For you, [0.4,0.4,0.2,0] should have lesser uncertainty than [0.3, 0.3, 0.3, 0.1], right? i.e., no. of maximal entries matters, right? – Ashok Aug 2 '12 at 14:34

1 Answer

A linear function would not have some essential properties reasonably required of such a measure of uncertainty. Consider, as in your examples, distributions $[p, 1-p]$, with $0 \le p \le 1$, on a two-point sample space, and let $h(p)$ be the measure of uncertainty embodied by such a distribution. Among the basic requirements of such a measure are that both $h(0) < h(1/2)$ and $h(1) < h(1/2)$ -- because a probability $p$ equal to $0$ or $1$ corresponds to less uncertainty than does any other value of $p$ -- but this is impossible if $h(p)$ is linear in $p$, or indeed if $h(p)$ is monotonic in $p$. Standard axiomatic developments lead to the concave function $h(p) = - p \log p - (1-p) \log (1-p)$.

-
Yes, you're right. It was my mistake, I didn't mean linearity in the whole space but in $[0, \frac{1}{2}]$ and $[\frac{1}{2}, 1]$, in a way that also verifies the property that you just stated. (So, in the first interval it will monotonically increasing and in the second one, monotonically decreasing). – synack Aug 2 '12 at 13:33
So, you could imagine the graph of the function that I propose similar to $-|x|$. – synack Aug 2 '12 at 13:44