Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

In the following integral, p(x) and q(x) are probability distributions. Can you help me to determine in what situation this integral is equal to infinity. For example, I think that such a situations is when only p(x) has an infinite peak.

$$\int_{-\infty}^{\infty}\{\log\frac{p(x)}{q(x)}\}p(x)dx$$

Thank you very much!

share|improve this question
    
Thank you for the edit! –  Marco Feb 8 '11 at 9:05
add comment

1 Answer 1

The integral that you have is the Kullback-Leibler divergence between distributions P and Q, $D_{KL}(P \parallel Q)$. This divergence is roughly a kind of a "distance" between the two distributions. The reason "distance" is in quotes is because this divergence is not symmetric and is hence not a metric. However, a useful way to think about the divergence $D_{KL}(P \parallel Q)$ is that it is the penalty paid for mistaking distribution $P$ as distribution $Q$. This statement can be made precise using information theory. If $D_{KL}(P \parallel Q)$ is infinity, it is because the two distributions are quite unlike each other that you incur an infinite penalty for mistaking $P$ as $Q$. This can happen for instance when distribution $P$ can produce values that distribution $Q$ can never do - in this case, mistaking $P$ as $Q$ is indeed a grievous error. I will leave it to you to interpret this in terms of the integral above to derive the condition under which the divergence is infinite.

Update: Adding more information in response to Marco's comment below. It got too unwieldy to be left as a comment: Given any $M>0$ and any distribution $P$, we can find $Q$ such that $D_{KL}(P \parallel Q) > M$. But note that as $M$ grows, we need to adaptively change $Q$ to make sure the divergence grows larger than $M$. This is different from saying $D_{KL}(P \parallel Q) = \infty$ for a given $P$, $Q$ which is what the question is stating. I think this can happen only if the support of $P$ includes points not in the support of $Q$.

share|improve this answer
    
Thank you, that interpretation helps. However, let us assume that p is absolutely continuous with respect to q, meaning that the support of p is a subset of the support of q. So distribution p cannot produce values that distribution q can never do. However, I think it is not enough to ensure finite values... –  Marco Feb 8 '11 at 10:04
    
What do you think of this? Suppose, for example, that p and q have the same support, (0, infinity). If q(x) --> 0 much more quickly than p(x) --> 0 then the integral is infinity. Thx –  Marco Feb 8 '11 at 10:39
    
Thank you very much for the update. I still need to think on it, but I am more and more convinced! Thank you! –  Marco Feb 9 '11 at 10:01
    
@Marco: See also mjqxxxx's explanation here: math.stackexchange.com/questions/21190/… –  Dinesh Feb 9 '11 at 17:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.