Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

From Wikipedia

In probability theory, the total variation distance between two probability measures $P$ and $Q$ on a sigma-algebra $F$ is $$ \sup\left\{\,\left|P(A)-Q(A)\right| : A\in F\,\right\}. $$ Informally, this is the largest possible difference between the probabilities that the two probability distributions can assign to the same event.

For a finite alphabet we can write $$ \delta(P,Q) = \frac 1 2 \sum_x \left| P(x) - Q(x) \right|\;. $$ Sometimes the statistical distance between two probability distributions is also defined without the division by two.

I was wondering if there is some particular consideration when having that $\frac 1 2$ for the finite case, while not in the general case? My understanding of this total variation distance/metric is that it is induced from upper variation of the whole set(which is a norm if I am correct). From there, I can't see the need of dividing by 2.

Also in the finite case, why not define similarly in terms of $\sup$ over $A \in F$?

Thanks and regards!

share|cite|improve this question
I don't understand what your question is. I suspect you've misinterpreted what that Wikipedia page (I strongly hesitate calling it an article) is saying. (Maybe the confusion arises having from the horrendous disconnect in the notation on that page.) Plus, the last result generalizes easily. – cardinal Jan 7 '12 at 20:18
Also, see the question and comments here:… – cardinal Jan 7 '12 at 20:21
@cardinal: Thanks! My questions are why there are inconsistency between the definition for the general case and the one for the discrete probability measures? One point is whether to divide by 2, and the other is whether to take sup over the sigma algebra. – Tim Jan 7 '12 at 20:26
In the finite case the two expressions are equivalent, though the notation on the Wikipedia page doesn't make that obvious. In other words, in the discrete case, $$\delta(P,Q) = \sup\{|P(A) - Q(A)|: A \in F\} = \frac{1}{2} \sum_x |P(x)-Q(x)|\>. $$ Can you supply a simple proof? See the hint in my comment to the above-linked question and define an appropriately analogous set $A$. – cardinal Jan 7 '12 at 20:27
@cardinal: The sup is achieved at either $A=\{x: P(x) \geq Q(x) \}$ or $A^c$, and $P(A)-Q(A) + P(A^c)-Q(A^c) = 0$. So the sup is achieved at both $A=\{x: P(x) \geq Q(x) \}$ and $A^c$. So ... Am I right? – Tim Jan 7 '12 at 20:46
up vote 5 down vote accepted

It is not a matter of adding a factor of $\frac{1}{2}$ in the finite case. The second expression is a sum over all elements of the underlying set, while the first expression is not a sum, but a sup over all events in the space. The reason for the $\frac{1}{2}$ in the second expression is that it can be proved that in the finite case, the two quantities are equal. See for example Proposition 4.2 on page 48 of Markov chains and mixing times by Levin, Peres, and Wilmer. I do not know the full extent of analogies to the second expression for cases when the underlying set is infinite, but the sum would have to become an integral. See cardinal's comments for more information.

share|cite|improve this answer
Yes, Jonas, there are easy extensions in the case of absolutely continuous distributions. See my comment at the link. – cardinal Jan 7 '12 at 20:38
@cardinal: Thank you. – Jonas Meyer Jan 7 '12 at 20:40

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.