Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

We know that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon's source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length assignments $\{l_i\}$ satisfying the Kraft inequality $\sum 2^{-l_i}\le 1$.

Also $H(p)=\log \frac{1}{p}$ is additive in the following sense. If $E$ and $F$ are two independent events with probabilities $p$ and $q$ respectively, then $H(pq)=H(p)+H(q)$.

As far as I know, mainly for these two reasons $H(p)=\log \frac{1}{p}$ is considered as a measure of information contained in a random event $E$ with probability $p>0$.

On the other hand, if we average the exponentiated lengths, $\sum p_i2^{tl_i}, t>0$, subject to the same Kraft inequality constraints, the optimal solution is $l_i=\log \frac{1}{p_i'}$ where $p_i'=\frac{p_i^{\alpha}}{\sum_k p_k^{\alpha}}, \alpha=\frac{1}{1+t}$, called Campbell's problem.

Now $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ is also additive in the sense that $H_{\alpha}(p_i p_j)=H_{\alpha}(p_i)+H_{\alpha}(p_j)$. Moreover $H_{\alpha}(1)=0$ as in the case of Shannon's measure.

Also note that, when $\alpha=1$, $H_1(p_i)=\log \frac{1}{p_i}$ we get back Shannon's measure.

My question is, are these reasons suffice to call $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ a (generalized) measure of information?

I don't know whether the dependence of measure of information of an event also on the probabilities of the other events make sense.

share|improve this question
    
Your notation is a bit unconventional. Normally $H(p)$ is used for Shannon entropy of a probability distribution (and not for a single value "p") given by $H(p) = -\sum_ip_i\log p_i$. The additivity you've described is a deeper property of this entropy function, not just a property of the logarithm as you've mentioned. There are a number of reasons why $H(p) = -\sum_ip_i\log p_i$ is used as the entropy function and not anything else. Refer to maths.gla.ac.uk/~tl/Renyi.pdf for the details. Hope this helps. –  VSJ Dec 26 '11 at 23:31
    
@VSJ Thanks for the response. $H(p)$ is also used to denote just $\log(1/p)$. If you have a look at the book On measures of information and their characterizations by Aczel and Daroczy, Chapter 0, section 2, you would notice that. It is proved in that section that $H(p)=\log(1/p)$ is the only function satisfying i) $H(p)\ge 0$, ii) $H(pq)=H(p)+H(q)$ and iii) $H(1/2)=1$. They say that for this reason $\log(1/p)$ is considered as a measure of information. My curiosity is whether $\log(1/p_i')$ can also be considered as a measure of information as it has the crucial additivity. –  Ashok Dec 27 '11 at 5:37
    
Also note that $\log(1/p_i')$ makes sense only when all the probabilities $p_1,\dots,p_n$ concerned with an experiment are considered. In that book they make a crucial assumption that the measure of information of an event depends only upon the probability of the event considered. But in the case of $\log(1/p_i')$, it depends probabilities of all events possible except when $\alpha=1$. –  Ashok Dec 27 '11 at 5:45
    
Aslo, please remember that I am not talking about the average here. My question is the following. An experiment is performed. $E_1,\dots,E_n$ are the possible disjoint events with respective probabilities $p_1,\dots,p_n$. Can we regard $\log(1/p_i')$ as a measure of information associated with $E_i$ as this answers a source compression problem and has the additivity property? –  Ashok Dec 27 '11 at 5:57

1 Answer 1

That's exactly the extension known as Renyi entropy with a normalization factor of $\frac{1}{1-\alpha}$.

$$H_\alpha(X) = \frac{1}{1-\alpha}\log\Bigg(\sum_{i=1}^n p_i^\alpha\Bigg)$$

  • Rényi, Alfréd (1961). "On measures of information and entropy". Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.