We know that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon's source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length assignments $\{l_i\}$ satisfying the Kraft inequality $\sum 2^{-l_i}\le 1$.

Also $H(p)=\log \frac{1}{p}$ is additive in the following sense. If $E$ and $F$ are two independent events with probabilities $p$ and $q$ respectively, then $H(pq)=H(p)+H(q)$.

As far as I know, mainly for these two reasons $H(p)=\log \frac{1}{p}$ is considered as a measure of information contained in a random event $E$ with probability $p>0$.

On the other hand, if we average the exponentiated lengths, $\sum p_i2^{tl_i}, t>0$, subject to the same Kraft inequality constraints, the optimal solution is $l_i=\log \frac{1}{p_i'}$ where $p_i'=\frac{p_i^{\alpha}}{\sum_k p_k^{\alpha}}, \alpha=\frac{1}{1+t}$, called Campbell's problem.

Now $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ is also additive in the sense that $H_{\alpha}(p_i p_j)=H_{\alpha}(p_i)+H_{\alpha}(p_j)$. Moreover $H_{\alpha}(1)=0$ as in the case of Shannon's measure.

Also note that, when $\alpha=1$, $H_1(p_i)=\log \frac{1}{p_i}$ we get back Shannon's measure.

My question is, are these reasons suffice to call $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ a (generalized) measure of information?

I don't know whether the dependence of measure of information of an event also on the probabilities of the other events make sense.

  • 1
    $\begingroup$ Your notation is a bit unconventional. Normally $H(p)$ is used for Shannon entropy of a probability distribution (and not for a single value "p") given by $H(p) = -\sum_ip_i\log p_i$. The additivity you've described is a deeper property of this entropy function, not just a property of the logarithm as you've mentioned. There are a number of reasons why $H(p) = -\sum_ip_i\log p_i$ is used as the entropy function and not anything else. Refer to maths.gla.ac.uk/~tl/Renyi.pdf for the details. Hope this helps. $\endgroup$ – VSJ Dec 26 '11 at 23:31
  • $\begingroup$ @VSJ Thanks for the response. $H(p)$ is also used to denote just $\log(1/p)$. If you have a look at the book On measures of information and their characterizations by Aczel and Daroczy, Chapter 0, section 2, you would notice that. It is proved in that section that $H(p)=\log(1/p)$ is the only function satisfying i) $H(p)\ge 0$, ii) $H(pq)=H(p)+H(q)$ and iii) $H(1/2)=1$. They say that for this reason $\log(1/p)$ is considered as a measure of information. My curiosity is whether $\log(1/p_i')$ can also be considered as a measure of information as it has the crucial additivity. $\endgroup$ – Ashok Dec 27 '11 at 5:37
  • $\begingroup$ Also note that $\log(1/p_i')$ makes sense only when all the probabilities $p_1,\dots,p_n$ concerned with an experiment are considered. In that book they make a crucial assumption that the measure of information of an event depends only upon the probability of the event considered. But in the case of $\log(1/p_i')$, it depends probabilities of all events possible except when $\alpha=1$. $\endgroup$ – Ashok Dec 27 '11 at 5:45
  • $\begingroup$ Aslo, please remember that I am not talking about the average here. My question is the following. An experiment is performed. $E_1,\dots,E_n$ are the possible disjoint events with respective probabilities $p_1,\dots,p_n$. Can we regard $\log(1/p_i')$ as a measure of information associated with $E_i$ as this answers a source compression problem and has the additivity property? $\endgroup$ – Ashok Dec 27 '11 at 5:57
  • $\begingroup$ @Ashok I was wondering, I have never heard of Shannon's source compression problem. However, I am sure its a very interesting theory and wanted to learn more about it. I was wondering, if you knew a good source to learn about it? textbook, blog, lecture note, website, etc? Thanks in advance! $\endgroup$ – Pinocchio Nov 9 '14 at 17:28

That's exactly the extension known as Renyi entropy with a normalization factor of $\frac{1}{1-\alpha}$.

$$H_\alpha(X) = \frac{1}{1-\alpha}\log\Bigg(\sum_{i=1}^n p_i^\alpha\Bigg)$$

  • Rényi, Alfréd (1961). "On measures of information and entropy". Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561.
  • $\begingroup$ //That's exactly the extension known as Renyi entropy with a normalization factor of...// Which one do you mean? $\endgroup$ – Ashok Nov 11 '14 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.