What is the intuition of why convergence in distribution does not imply convergence in probability

Question

For me its very counter intuitive (that convergence in Probability and Distribution are not the same), because, conceptually, if two random variables have the same distribution, then they should be considered the same random variable. I know this is wrong. I am not sure why but its wrong (hence the question). Maybe, does it mean they are the same (isomorphic) w.r.t. to their pdf/cdf/pmf? Anyway, I want to understand why I am wrong and fix my mathematical intuition.

The reason I am complaining is because I am completely aware of a counter example. In fact I will provide it in the appendix of this question, however, even after seeing the counter example, I am not sure I could have come up with it. In fact even after seeing a counter example, it still feels wrong. I was wondering if someone could provide a way of how one would come up with such a counter example (or maybe a counter example that is much more obvious), or maybe explain to me why it should be obvious that one can come up with such a counter example.

---

Appendix:

Recall the definitions of convergence in distribution and convergence in probability:

Convergence in distribution (D): $$ \lim_{n \rightarrow \infty} | F_{X_n}(x) - F(x)| = 0, \forall x \in X$$

Convergence in probability (P) for sufficiently large $N$ and $\forall \epsilon > 0$:

$$ \lim_{n \rightarrow \infty} Pr[ | X_n - X | \geq \epsilon]= 0$$

i.e. if one goes sufficiently far the sequence of random variables, then, $X_n$ will converge to $X$ (in probability).

I will show the counter example for:

$$D \not\implies P$$

Consider a Bernoulli random variable $X$ with probability of success $\frac{1}{2}$. Now define $X_n$ to be equal to $X$, i.e.

$$X_n = X$$

Therefore, obviously since they are the same r.v. they have the same distribution. Now define:

$$Y = 1 - X$$

Obviously, $Y$ has the same distribution as $X$ and hence, also as $X_n$. So $X_N$ converges in distribution to $Y$.

But, do they converge in probability?

If they do then:

$$ \lim_{n \rightarrow \infty} Pr[ | X_n - Y | \geq \epsilon]= 0$$

i.e. we want their difference to be larger than $\epsilon$ very rarely (in fact we want that event to have zero probability zero as n goes to infinity).

Let's figure out the distance between these two random variables $X_n$ and $Y$:

$$| X_n - Y | = |X_N - (1 - X)| = |X - 1 + X| = |2X - 1| = 1$$

Therefore, their distance is always 1, no matter what. We want their distance to be very far for any $\epsilon$, but notice that for the choice of $\epsilon = \frac{1}{2}$, that the distance between $| X_n - Y | = 1$ is indeed greater than the tolerable threshold (i.e. $1 = | X_n - Y | > \epsilon = \frac{1}{2} $ with probability 1). So for sure their distance is more than $\epsilon$. It doesn't matter what value of $n$ we choose (i.e. it doesn't matter how far we go down the sequence), they are for sure too far apart. More precisely:

$$ \lim_{n \rightarrow \infty}Pr[ | X_n - Y | \geq \epsilon ] = Pr[ 1 \geq \frac{1}{2} ] = 1 $$

which is the opposite of what we need for convergence in probability. End of counter example.

Answer 1

First, in convergence in distribution, the random variables can be defined on different spaces, whereas convergence in probability requires them all to be defined on the same space (unless the limiting 'random variable' is degenerate).

Second, even artificially requiring that all random variables are defined on the same space, it is easy to find examples of sequences of random variables that converge in distribution, but not in probability. Here is (perhaps) the most trivial example.

Let the sample space have only two points $a$ and $b$, each with probability $1/2.$ Let all $X_n$ be the same with $X_n(a) = 0$ and $X_n(b) = 1$; let $Y$ be defined as $Y(a) = 1$ and $Y(b) = 0$. (Notice the crucial switch.) Then $F_{X_n}(x)$ has jumps of $1/2$ at $x = 0$ and $x = 1.$ Also, the CDF of $Y$ is exactly the same function. So $F_{X_n}(x) \equiv F_Y(x)$ and $|F_{X_n}(x) - F_Y(x)| \equiv 0$, making convergence in distribution trivial.

By contrast, $|X_n(a) - Y(a)| = 1$ and $|X_n(b) - Y(b)| = 1,$ so $P(|X_n - Y| = 1) = 1,$ and convergence in probability is impossible.

Finally, for convergence to a constant, the two concepts can be interpreted as equivalent.

Answer 2

Your example consists of the distribution $(X,Y) \in_R \{(1,0),(0,1)\}$. Considered individually, $X$ and $Y$ are the "same" random variable. However, they are not independent, so $X$ and $Y$ are not the same as two independent copies of $X$.

There are other simple counterexamples out there, such as the following. Let $X$ and $X_1,X_2,\ldots$ be independent standard Gaussians. They all have the same law, so the sequence $X_i$ convergence to $X$ in distribution. But clearly it doesn't converge to $X$ in probability. Here everything is independent, so the problem is different: in fact, the problem is that the sequence $X_i$ is independent of $X$.

Perhaps it's best to consider an example in which $X_i$ does converge to $X$ in probability. Let $X$ be any (reasonable) random variable, let $Z_i$ be independent standard Gaussians, let $Y_i = X + Z_i$, and let $X_i = (Y_1 + \cdots + Y_i)/i$. Under mild conditions, the sequence $X_i$ converges to $X$. The picture here is that the $X_i$ are noisy versions of $X$, but the noise tends to zero. This just doesn't happen in the two counterexamples (yours and mine).

Answer 3

Lim P( Xn(w)-X(w) <= E ) = 1 for all w of the population. We can induce the same distribution using two different random variables. Your example shows it good. If this random variables are different then, p(x=1)=p(y=1) whereas x^-1(1) is a different set of the population from y^-1(1). This means convergence in distribution is weakening the hypothesis needed for a succession of random variables to converge in probability. When imagining this, just associate a set of w's to each point of the domain of the distribution of the random variable.