How close is the sample mean $\mu_N$ for $N$ random variables to the sample mean $\mu_{N-1}$ for $N-1$ random variables? Define $\mu_{N}$ to be the sample mean for $N$ i.i.d. random variables that each have mean $\mu$ and variance $\sigma^2$:
$$
\mu_{N} := \frac{1}{N} \sum_{i=1}^N X_i.
$$
Now consider the sample mean if we removed one of these random variables. That is consider, $\mu_{N-1}$ where w.l.o.g. we remove the last random variable from the sum:
$$
\mu_{N-1} := \frac{1}{N-1} \sum_{i=1}^{N-1} X_i.
$$
I am interested in describing how close $\mu_{N-1}$ is to $\mu_{N}$ in an appropriate sense..it is this 'sense' that I am unsure of as I haven't been working with probability that long.
Subtracting $\mu_{N-1}$ from $\mu_{N}$ we have
\begin{align}
\mu_{N} - \mu_{N-1} &= \frac{1}{N} \sum_{i=1}^N X_i - \frac{1}{N-1} \sum_{i=1}^{N-1} X_i \\
& = \frac{1}{N}X_N + \frac{N-1}{N}\frac{1}{N-1} \sum_{i=1}^{N-1} X_i - \frac{1}{N-1} \sum_{i=1}^{N-1} X_i \\
& = \frac{1}{N}X_N + \bigg(\frac{N-1}{N}-1\bigg)\frac{1}{N-1} \sum_{i=1}^{N-1} X_i.
\end{align}
So as $N\to \infty$, heurstically it 'looks like' both terms go to zero. But as the $X_i$ are random variables I imagine we need to set some conditions on them for this to hold. For example, we could be unlucky and draw $X_N$'s that are sufficiently large such that $\frac{1}{N} X_N$ does not go to zero as $N \to \infty$.
I feel like the variance $\sigma^2$ could have an effect on the convergence but I'm not sure.
So my questions are

*

*In the finite sample case can we say something more informative than what I have already derived above regarding how close $\mu_N$ is to $\mu_{N-1}$?

*What conditions are needed on the $X_i$ to ensure $\mu_{N} - \mu_{N-1}$ converges to zero as $N \to \infty$?

*What is an appropriate sense of convergence for $\mu_{N} - \mu_{N-1}$? I am thinking maybe we can say it converges with 'high probability'. Most importantly, is it possible to find an exact rate of convergence?

 A: You need to read about the different concepts of convergence of random variables.
In this case, calling $Z_n =  \frac{1}{N} \sum_{i=1}^N X_i - \frac{1}{N-1} \sum_{i=1}^{N-1} X_i$, you want to know if and (in what sense ) $Z_n$ converges to zero. Notice that $Z_n$ is a random variable.
Given that $X_i$ are iid with finite mean and variance, you can deduce that $E[Z_n]=0$ and $\sigma_{Z_n}^2 =   \frac{1}{n(n+1)}\sigma_{X}^2$.
This says that $\sigma_{Z_n}^2 \to 0$ (and quite quick).
Then you can apply  Chebyshev's_inequality : for any $\epsilon >0$
$$P(|Z_n| > \epsilon) \le \frac{1}{\epsilon}\sigma_{Z_n}^2=\frac{1}{\epsilon \, n(n+1)}\sigma_{X}^2 < \frac{\sigma_{X}^2}{\epsilon \, n^2} $$
Then $P(|Z_n| > \epsilon) \to 0$ as $n\to \infty$
The above amounts to prove convergence in probability from convergence in $L^2$-norm. These are quite strong measures of convergence for random variables, though not so strong as almost sure convergence.
A: 
$Z_n\to 0, n\to\infty,$ almost surely iff $\mathrm{E}[|X_1|]<\infty$.

Proof. Sufficiency: by the Kolmogorov SLLN, $\mu_n \to 0$ and $\mu_{n-1} \to 0$, $n\to\infty$, almost surely.
The necessity is proved similarly to the proof of SLLN: since $\mu_{n-1}$ and $X_n$ are independent, it is clear that the both terms in the expression you wrote for $\mu_n - \mu_{n-1}$ must converge to $0$, in particular, $X_n/n \to 0, n\to\infty$, a.s. Then, with probability $1$, $|X_n|/n > 1$ only finitely many times. By the reverse Borel–Cantelli lemma, we then must have
$$
\sum_{n=1}^\infty \mathrm P(|X_n|>n) = \sum_{n=1}^\infty \mathrm P(|X_1|>n)<\infty,
$$
which is equivalent to integrability of $X_1$.
Using the strong law of large numbers for non-integrable random variables (I can elaborate if you wish), it is also possible to prove that $Z_n \to 0$ in probability provided that $\mathrm{E} [|X_1|^{1/2}]<\infty$, though I have a doubt that this is also a necessary condition (it is necessary for the second term in your expression to vanish almost surely; it is also possible to formulate a necessary and sufficient conditions in terms of behavior of characteristic function at zero).
