# How do concepts such as limits work in probability theory, as opposed to calculus?

When I am flipping a fair coin and say that as the number of trials approaches $$\infty$$ the number of heads approaches $$50\%$$, what do I really mean?

Intuitively, I would associate it with the concept of a limit, as used in calculus:

$$\lim_{t \to \infty} \left(\frac{H}{H+T}\right)=0.5 \\ \text{Where t = the number of trials, H = the number of heads, and T = the number of tails}$$

However, this intuition seems to break down when I use the formal definition of a limit:

$$\lim_{t \to \infty} \left(\frac{H}{H+T}\right)=0.5 \text{ if and only if}\\ \text{for every \varepsilon>0, there exists N>0 such that for all t} \\ \text{if t>N then |0.5-\frac{H}{H+T}|<\varepsilon}$$

Well I don't know if there will be $$N > 0$$ that satisifes this definition! It all depends on what comes up. However many times I flip the coin, $$\frac{H}{H+T}$$ might just equal $$0$$. So how might I define terms such as "approaches" in probability theory if the conventional definition does not work?

Edit: User nicomezi has pointed out that this a huge topic. Therefore, I will accept even a very short introduction to this subject as an answer.

• This is a huge topic. You have already been given a (good) first insight about it but a decent answer would need a book, or a least a few chapters. – nicomezi Jun 4 at 7:38
• @nicomezi Thank you for responding. I did not expect this to be such a large topic. I don't mind if the answers given are only a very short introduction to the groundwork of probability theory. I'll edit my question to indicate this. – Joe Jun 4 at 8:28
• $\delta$ is usually meant to be a small number. For $\lim_{t\to\infty}$, we often say "... there exists $N>0$ such that for all $t$, if $t>N$ then ..." to suggest that $N$ is a large number. – Teepeemm Jun 4 at 19:37
• @Teepeemm Thank you. I did not know that. I will change my question accordingly. – Joe Jun 4 at 21:13

There are several notions of convergence in probability.

Your example is an instance of the law of large numbers which has a weak form and a strong form.

The weak form states that $$H/(H+T)$$ "converges in probability" to $$0.5$$. Formally, for any $$\epsilon > 0$$, $$\lim_{t \to \infty}P\left(\left|\frac{H}{H+T} - 0.5\right| > \epsilon\right) = 0.$$

The strong form states that $$H/(H+T)$$ "converges almost surely" to $$0.5$$. Formally, $$P\left(\lim_{t \to \infty} \frac{H}{H+T} = 0.5\right) = 1.$$ Almost sure convergence implies convergence in probability, hence the strong law implies the weak law (but is harder to prove).

Response to comment:

If you imagine flipping a coin many many times, then you can keep track of the sequence $$\frac{H}{H+T}$$ as $$t$$ increases. In your words, this sequence is random, since it "depends on what comes up," so in different parallel universes this sequence will be different. You are correct that it is possible that you always flip tails, in which case the sequence is $$0,0,\ldots$$. But given a particular sequence, you simply have a sequence of real numbers, so the limit in $$\lim_{t \to \infty} \frac{H}{H+T}$$ is the usual limit you are familiar with (which may not exist for some sequences). The law of large numbers states that with probability $$1$$, this sequence not only has a limit, but that limit is $$0.5$$. So scenarios like the one you mentioned (always flipping tails) happen with probability $$0$$.

• Thank you very much. That's very clever! So you take the limit of a probability (or the probability of a limit), meaning that the randomness is inherently baked into our definition of convergence. The weak form of the law of large numbers makes complete intuitive sense to me (although I cannot fathom the proof). The strong form is a bit more confusing, but is probably beyond the realm of my understanding anyway. Just to clarify, when you use '$\lim$' in the formal statement of the law of large numbers, does this use the same definition of a limit as I used? – Joe Jun 4 at 8:24
• @Joe See my updated answer – angryavian Jun 4 at 17:21

In probability, you work with distributions. When you increase the number of trials and take the average, you get a distribution that tends to a Gaussian, with a narrower variance.

The conclusions that you can draw are based on this probabilistic inference. For an infinite number of drawings, the distribution tends to a Dirac delta. 