2
$\begingroup$

Total variation distance is a measure for comparing two probability distributions (assuming that these are unit vectors in a finite space- where basis corresponds to the sample space ($\omega$)). I know a distance measure need to obey triangle inequality and it should satisfy that orthogonal vectors have maximum distance and the same distributions should have distance $0$. Others should like between these two. I completely don't understand why the $L^1$ norm is chosen for measuring the distance between these vectors (prob. distributions). I also want to know why it is exactly defined the way it is. $TV(P_1,P_2) = \frac{1}{2}\sum_{x \in \omega} \mid {P_1(x)-P_2(x) \mid}$

$\endgroup$
4
  • $\begingroup$ Yes. That's what but here $f,g$ are probability distributions. I don't understand why $L^1$ is used and why 1/2 comes in the way? $\endgroup$
    – am_rf24
    Oct 30 '19 at 20:20
  • $\begingroup$ The total variation distance of two probability measures is usually defined in terms of the sup norm: en.wikipedia.org/wiki/… $\endgroup$
    – amsmath
    Oct 30 '19 at 20:22
  • $\begingroup$ "When the set($\omega$) is countable, the total variation distance is related to the $L^1$ norm by the identity." So, this follows by doing some math. But what is the intuitive explanation for this? $\endgroup$
    – am_rf24
    Oct 30 '19 at 20:36
  • $\begingroup$ See Proposition 4.2 in pages.uoregon.edu/dlevin/MARKOV/markovmixing.pdf $\endgroup$
    – amsmath
    Oct 30 '19 at 20:42
1
$\begingroup$

The TV distance is measureing exactly what you want: The maximal difference between the assignments of probabilities of two probability distributions P and Q. And it is hence defined as $$TV(P,Q)=sup_{A\subset\Omega}|P(A)-P(Q)|$$

Now, as is shown in Proposition 4.2 here your last equation $$TV(P,Q)=\frac{1}{2}||P-Q||_1$$ is true (in countable prob. spaces). I will not redo the proof here, you can get a glimpse using the characterizations am_rf24 writes about in his answer. But I can give you some intuition: Although the definition of the TV distance seems closely resembling the definition of the infinity norm on vectors, it is actually subtly different. Note that the TV disctance is defined over events aka subsets of $\Omega$ while the infinity norm is over elements of $\Omega$. So to conclude: The one norm is rather coincidentially equivalent to the TV distance and is not forcedly choosen as the norm of choice. But because it is a norm we luckily do not need to show that the TV distance is a norm.

(Another remark: the orthogonality criterion you mention is not really a thing for a norm, as you can have norms without having a scalar product ;) )

$\endgroup$
0
$\begingroup$

The two observations for characterizing the total variation distance using the two definitions are $\\$: $ 1. \max{P_1(A)-P_2(A)} = \max{P_2(\omega-A)-P_1(\omega-A)} \\$ $2. P_1(\omega)= P_2(\omega) \implies P_1(A)- P_2(A) = P_2(B) -P_1(B) = \max_{A \subset \omega}( P_1(A)- P_2(A)) $

Now, using $ 1/2 \sum_{x}{\mid{P_1(x)-P_2(x)}\mid}= \frac{1}{2}\sum_{x:P_1(x)>P_2(x)}{{(P_1(x)-P_2(x))}}+ \frac{1}{2}\sum_{x:P_2(x)>P_1(x)}{{(P_2(x)-P_1(x))}}$ $ \qquad \qquad \qquad \qquad= \frac{1}{2}\max_{A \subset \omega}{(P_1(A)-P_2(A))} + \frac{1}{2} \max_{B \subset \omega}{(P_2(B)-P_1(B))}$
$ \qquad \qquad \qquad \qquad= \max_{A \subset \omega}{ \mid P_1(A)-P_2(A)\mid}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.