# Total variation distance, $L^1$ norm

Total variation distance is a measure for comparing two probability distributions (assuming that these are unit vectors in a finite space- where basis corresponds to the sample space ($$\omega$$)). I know a distance measure need to obey triangle inequality and it should satisfy that orthogonal vectors have maximum distance and the same distributions should have distance $$0$$. Others should like between these two. I completely don't understand why the $$L^1$$ norm is chosen for measuring the distance between these vectors (prob. distributions). I also want to know why it is exactly defined the way it is. $$TV(P_1,P_2) = \frac{1}{2}\sum_{x \in \omega} \mid {P_1(x)-P_2(x) \mid}$$

• Yes. That's what but here $f,g$ are probability distributions. I don't understand why $L^1$ is used and why 1/2 comes in the way? Oct 30 '19 at 20:20
• The total variation distance of two probability measures is usually defined in terms of the sup norm: en.wikipedia.org/wiki/… Oct 30 '19 at 20:22
• "When the set($\omega$) is countable, the total variation distance is related to the $L^1$ norm by the identity." So, this follows by doing some math. But what is the intuitive explanation for this? Oct 30 '19 at 20:36
• See Proposition 4.2 in pages.uoregon.edu/dlevin/MARKOV/markovmixing.pdf Oct 30 '19 at 20:42

The TV distance is measureing exactly what you want: The maximal difference between the assignments of probabilities of two probability distributions P and Q. And it is hence defined as $$TV(P,Q)=sup_{A\subset\Omega}|P(A)-P(Q)|$$
Now, as is shown in Proposition 4.2 here your last equation $$TV(P,Q)=\frac{1}{2}||P-Q||_1$$ is true (in countable prob. spaces). I will not redo the proof here, you can get a glimpse using the characterizations am_rf24 writes about in his answer. But I can give you some intuition: Although the definition of the TV distance seems closely resembling the definition of the infinity norm on vectors, it is actually subtly different. Note that the TV disctance is defined over events aka subsets of $$\Omega$$ while the infinity norm is over elements of $$\Omega$$. So to conclude: The one norm is rather coincidentially equivalent to the TV distance and is not forcedly choosen as the norm of choice. But because it is a norm we luckily do not need to show that the TV distance is a norm.
The two observations for characterizing the total variation distance using the two definitions are $$\\$$: $$1. \max{P_1(A)-P_2(A)} = \max{P_2(\omega-A)-P_1(\omega-A)} \\$$ $$2. P_1(\omega)= P_2(\omega) \implies P_1(A)- P_2(A) = P_2(B) -P_1(B) = \max_{A \subset \omega}( P_1(A)- P_2(A))$$
Now, using $$1/2 \sum_{x}{\mid{P_1(x)-P_2(x)}\mid}= \frac{1}{2}\sum_{x:P_1(x)>P_2(x)}{{(P_1(x)-P_2(x))}}+ \frac{1}{2}\sum_{x:P_2(x)>P_1(x)}{{(P_2(x)-P_1(x))}}$$ $$\qquad \qquad \qquad \qquad= \frac{1}{2}\max_{A \subset \omega}{(P_1(A)-P_2(A))} + \frac{1}{2} \max_{B \subset \omega}{(P_2(B)-P_1(B))}$$
$$\qquad \qquad \qquad \qquad= \max_{A \subset \omega}{ \mid P_1(A)-P_2(A)\mid}$$