Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am encountering a situation where I cannot calculate exact sum of a seris of logorithms in calculating entropy. Suppose we have a series of numbers $p_i$ and we want to calculate $\sum_ilog(p_i)$, we should multiply $p_i$ together then take the log, or just take the log of each $p_i$ and calculate the summation over them?

The $p_i$ satisfies that $\sum_i p_i = 1$ and each $p_i$ is between 0 and 1.

Things are, we multiply them together, seems I will encounter the loss of precision and calculate the multiplication will accumulate the loss of precision. But when doing log apart, then each log seems to have loss of precisions.

share|cite|improve this question
You would actually want to find $\sum_i p_i\log p_i$, know? In standard texts, they always find $\log$ of each $p_i$ and multiply by $p_i$ and sum it up. But I do not know which is better and why. – Ashok Oct 1 '12 at 12:44

Using standard computer floating point you will have errors at every step along the way, but they shouldn't be too bad. Your $p_i$'s probably cannot be represented exactly, so even $p_1+p_2$ has some error associated. In 64 bit floats, this amounts to a relative error of order $2^{-53}$, so even a lot of those rarely amount to much. It was worse in the 32 bit days. The log function will have errors, too, but if you use a decent library function it shouldn't be much worse. The risk when you multiply comes if you overflow the available range, but this is unlikely too.

The real problem usually comes when you subtract two nearly equal quantities. In your case the terms all have the same sign, so this is not a problem. If your terms cover several orders of magnitude, it is better to add the small ones first.

share|cite|improve this answer
Related:… – leonbloy Oct 31 '14 at 10:22

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.