# measure of information

We know that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon's source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length assignments $\{l_i\}$ satisfying the Kraft inequality $\sum 2^{-l_i}\le 1$.

Also $H(p)=\log \frac{1}{p}$ is additive in the following sense. If $E$ and $F$ are two independent events with probabilities $p$ and $q$ respectively, then $H(pq)=H(p)+H(q)$.

As far as I know, mainly for these two reasons $H(p)=\log \frac{1}{p}$ is considered as a measure of information contained in a random event $E$ with probability $p>0$.

On the other hand, if we average the exponentiated lengths, $\sum p_i2^{tl_i}, t>0$, subject to the same Kraft inequality constraints, the optimal solution is $l_i=\log \frac{1}{p_i'}$ where $p_i'=\frac{p_i^{\alpha}}{\sum_k p_k^{\alpha}}, \alpha=\frac{1}{1+t}$, called Campbell's problem.

Now $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ is also additive in the sense that $H_{\alpha}(p_i p_j)=H_{\alpha}(p_i)+H_{\alpha}(p_j)$. Moreover $H_{\alpha}(1)=0$ as in the case of Shannon's measure.

Also note that, when $\alpha=1$, $H_1(p_i)=\log \frac{1}{p_i}$ we get back Shannon's measure.

My question is, are these reasons suffice to call $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ a (generalized) measure of information?

I don't know whether the dependence of measure of information of an event also on the probabilities of the other events make sense.

-
Your notation is a bit unconventional. Normally $H(p)$ is used for Shannon entropy of a probability distribution (and not for a single value "p") given by $H(p) = -\sum_ip_i\log p_i$. The additivity you've described is a deeper property of this entropy function, not just a property of the logarithm as you've mentioned. There are a number of reasons why $H(p) = -\sum_ip_i\log p_i$ is used as the entropy function and not anything else. Refer to maths.gla.ac.uk/~tl/Renyi.pdf for the details. Hope this helps. – VSJ Dec 26 '11 at 23:31
@VSJ Thanks for the response. $H(p)$ is also used to denote just $\log(1/p)$. If you have a look at the book On measures of information and their characterizations by Aczel and Daroczy, Chapter 0, section 2, you would notice that. It is proved in that section that $H(p)=\log(1/p)$ is the only function satisfying i) $H(p)\ge 0$, ii) $H(pq)=H(p)+H(q)$ and iii) $H(1/2)=1$. They say that for this reason $\log(1/p)$ is considered as a measure of information. My curiosity is whether $\log(1/p_i')$ can also be considered as a measure of information as it has the crucial additivity. – Ashok Dec 27 '11 at 5:37
Also note that $\log(1/p_i')$ makes sense only when all the probabilities $p_1,\dots,p_n$ concerned with an experiment are considered. In that book they make a crucial assumption that the measure of information of an event depends only upon the probability of the event considered. But in the case of $\log(1/p_i')$, it depends probabilities of all events possible except when $\alpha=1$. – Ashok Dec 27 '11 at 5:45
Aslo, please remember that I am not talking about the average here. My question is the following. An experiment is performed. $E_1,\dots,E_n$ are the possible disjoint events with respective probabilities $p_1,\dots,p_n$. Can we regard $\log(1/p_i')$ as a measure of information associated with $E_i$ as this answers a source compression problem and has the additivity property? – Ashok Dec 27 '11 at 5:57
@Ashok I was wondering, I have never heard of Shannon's source compression problem. However, I am sure its a very interesting theory and wanted to learn more about it. I was wondering, if you knew a good source to learn about it? textbook, blog, lecture note, website, etc? Thanks in advance! – Pinocchio Nov 9 '14 at 17:28

That's exactly the extension known as Renyi entropy with a normalization factor of $\frac{1}{1-\alpha}$.
$$H_\alpha(X) = \frac{1}{1-\alpha}\log\Bigg(\sum_{i=1}^n p_i^\alpha\Bigg)$$