# Proving a sample mean converges in probability to the true mean

I have an i.i.d. sequence of observations $\{x_1, x_2,\ldots, x_n\}$ and I need to prove that $\frac{1}{n}\sum x_i$ converges in probability the "true mean", that it is consistent estimate of the mean.

But seems to me that I already have the true mean defined... and have nothing to show.

• Do you have the central limit theorem? Note that the true mean is the expectation of the distribution on $x_i$. This is not generall the same as $\frac{1}{n}\sum_i x_i$ – Ben Grossmann Mar 17 '14 at 16:55
• I prefer using the term "population mean". Maybe I should post a careful explanation of why that is so. – Michael Hardy Mar 17 '14 at 16:59
• Note that this is not true if the variance is infinite. – leonbloy Mar 18 '14 at 3:14

Use Chebyshev's inequality.

First recall Markov's inequality: If $\Pr(V\ge0)=1$ and $\mathbb E V=\alpha$ then $\Pr(V\ge x\alpha)\le\dfrac 1 x$. E.g. if incomes are non-negative, then no more than $1/15$ of the population can have more than $15$ times the average income, etc.

Apply this to $(X-\mu)^2$ where $\mu = \mathbb EX$ and $\sigma^2=\mathbb E((X-\mu)^2)<\infty$.

Markov's inequality then tells is that $\Pr((X-\mu)^2\ge x^2 \sigma^2)\le \dfrac 1 {x^2}$.

Consequently $\Pr\left(\dfrac{|X-\mu|}{\sigma}\ge x\right)\le \dfrac 1 {x^2}$. That is Chebyshev's inequality.

Let $\bar X = (X_1+\cdots+X_n)/n$, and recall that $\mathbb E \bar X=\mu$ and $\operatorname{var}(\bar X)=\sigma^2/n$. (This assumes $X_1,X_2,X_3,\ldots$ all have the same expected value and the same finite variance, and are uncorrelated. It is not necessary to assume they all have the same distribution, nor that they are independent, although those assumptions are certainly sufficient, since they are stronger than the assumptions we're making here.)

Applying Chebyshev's inequality to $\bar X$, we get $$\Pr\left(\frac{|\bar X-\mu|}{\sigma/\sqrt{n}}\ge x\right) \le \frac {1}{x^2},$$ so $$\Pr\left(|\bar X-\mu|\ge \frac{x\sigma}{\sqrt{n}}\right) \le \frac 1 {x^2}.$$ This holds if $x$ is any positive number at all; hence it holds when $x=\varepsilon\sqrt{n}/\sigma$, where $\varepsilon$ is any positive number. That gives us $$\Pr\left(|\bar X-\mu|\ge \varepsilon\right) \le \frac{\sigma^2}{\varepsilon^2 n}\to0\text{ as }n\to\infty.$$

All this works in cases where $\sigma^2<\infty$. In cases where that does not hold, there are counterexamples to the conclusion we're getting here, but I think some weaker assumptions than those we've made might be enough.

You need to distinguish between the two concepts: the "true" mean of the underlying distribution, i.e. $\mathbb{E}(X_1)$ and the sample mean $\frac{1}{n}\sum X_i$. The goal is to show that $\frac{1}{n}\sum X_i \rightarrow \mathbb{E}(X_1)$ in probability as $n$ tends to infinity.

• Yes, that's the goal. Is it achievable? – snoram Mar 17 '14 at 16:25
• Yes, what you should use is some Law of Large Numbers. Then it's straightforward. – Jacek Podlewski Mar 17 '14 at 16:27
• @JacekPodlewski : The statement to be proved here is itself the weak law of large numbers. If you use the weak law of large numbers to prove it, then that's circular reasoning. – Michael Hardy Mar 17 '14 at 17:38