It is frequently stated (in textbooks, on Wikipedia) that the "Law of large numbers" in mathematical probability theory is a statement about relative frequencies of occurrence of an event in a finite number of trials or that it "relates the axiomatic concept of probability to the statistical concept of frequency". Isn't this is a methodological mistake of ascribing an interpretation to a mathematical term, perhaps relying too much on the colorful language, that does not at all follow from how this term is mathematically defined? Recall the typical derivation of the WLLN:

Let $X_1, X_2, ..., X_n$ be a sequence of n independent and identically distributed random variables with the same finite mean $\mu$, and with variance $\sigma^2$ and let:


We have:

$E[\overline{X}] = \frac{E[X_1+...+X_n]}{n} = \frac{E[X_1]+...+E[X_n]}{n} = \frac{n\mu}{n} = \mu$ $Var[\overline{X}] = \frac{Var[X_1+...+X_n]}{n^2} = \frac{Var[X_1]+...+Var[X_n]}{n^2} = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}$

And from Chebyshev's inequality:

$P(|\overline{X}-\mu|>\epsilon) \le \frac{\sigma^2}{n\epsilon^2}$

And so X is said to converge in probability to $\mu$.

Now consider what is strictly speaking the meaning of this expression in the axiomatic framework it is derived in:

$P(|\overline{X}-\mu|>\epsilon) \le \frac{\sigma^2}{n\epsilon^2}$

$P()$, everywhere it occurs in the derivation, is known only to be a number satisfying Kolmogorov's axioms, so a number between 0 and 1, and so forth, but none of the axioms introduce any theoretical equivalent of the intuitive notion of frequency. If additional assumptions about $P()$ are not made, the sentence can obviously not be interpreted at all, but what is also important the theoretical mean $\mu$ is not necessarily the mean value in an infinite number of trials, $\overline{X}$ is not necessarily the mean value from n trials, and so forth. Consider an experiment of tossing a fair coin repeatedly - quite obviously, nothing in Kolmogorov's axioms enforces using 1/2 for the probability of heads, you could just as well use $1/\sqrt{\pi}$, yet the derivation continues to "work", except the meaning of the various variables is not in agreement with their intuitive interpretations. The $P()$ might still mean something, it might be a quantification of an absurd belief of mine, the mathematical derivation continues be true regardless, in the sense that as long as the initial $P()'s$ satisfy axioms, theorems about other $P()'s$ follow, and with Kolmogorov's axioms providing only weak constraints on and not a definition of $P()$, it's basically only symbol manipulation.

This "relative frequency" interpretation frequently given seems to rest on an additional assumption, and this assumption seems to be a form of the law of large numbers itself. Consider this fragment from Kolmogorov's Grundbegriffe on applying the results of probability theory to the real world:

We apply the theory of probability to the actual world of experiment in the following manner:


4) Under certain conditions, which we shall not discuss here, we may assume that the event A which may or may not occur under conditions S, is assigned a real number P(A) which has the following characteristics:

a) One can be practically certain that if the complex of conditions S is repeated a large number of times, n, then if m be the number of occurrences of event A, the ratio m/n will differ very slightly from P(A).

Which seems equivalent to introducing the weak law of large numbers in a particular, slightly different form, as an additional axiom.

Meanwhile, many reputable sources contain statements that seem completely in opposition to the above reasoning, for example Wikipedia:

It follows from the law of large numbers that the empirical probability of success in a series of Bernoulli trials will converge to the theoretical probability. For a Bernoulli random variable, the expected value is the theoretical probability of success, and the average of n such variables (assuming they are independent and identically distributed (i.i.d.)) is precisely the relative frequency.

This seem to be mistaken already in claiming that from a mathematical theorem anything can follow about empirical probability (the page on which defines it as the relative frequency in actual experiment), but there are many more subtle claims that technically also seem erroneous from the above considerations:

The LLN is important because it "guarantees" stable long-term results for the averages of random events.

Note that the Wikipedia article about LLN claims to be about the mathematical theorem, not about the empirical observation, which was also historically sometimes been called the LLN. It seems to me that LLN does nothing to "guarantee stable long-term results", for as stated above those stable long-term results have to be assumed in the first place for the terms occuring in the derivation to have the intuitive meaning we typically ascribe to them, not to mention something has to be done to at all interpret $P()$ in the first place. Another instance from Wikipedia:

According to the law of large numbers, if a large number of six-sided die are rolled, the average of their values (sometimes called the sample mean) is likely to be close to 3.5, with the precision increasing as more dice are rolled.

Does this really follow from the mathematical theorem? In my opinion, the interpretation of the theorem that is used here, rests on assuming this fact. There is a particularly vivid example in the "Treatise on probability" by Keynes of what happens when one follows the WLLN with even a slight deviation from this initial assumptions of p's being the relative frequencies in the limit of an infinite number of trials:

The following example from Czuber will be sufficient for the purpose of illustration. Czuber’s argument is as follows: In the period 1866–1877 there were registered in Austria

m = 4,311,076 male births

n = 4,052,193 female births

s = 8,363,269

for the succeeding period, 1877–1899, we are given only

m' = 6,533,961 male births;

what conclusion can we draw as to the number n of female births? We can conclude, according to Czuber, that the most probable value

n' = nm'/m = 6,141,587

and that there is a probability P = .9999779 that n will lie between the limits 6,118,361 and 6,164,813. It seems in plain opposition to good sense that on such evidence we should be able with practical certainty P = .9999779 = 1 − 1/45250 to estimate the number of female births within such narrow limits. And we see that the conditions laid down in § 11 have been flagrantly neglected. The number of cases, over which the prediction based on Bernoulli’s Theorem is to extend, actually exceeds the number of cases upon which the à priori probability has been based. It may be added that for the period, 1877–1894, the actual value of n did lie between the estimated limits, but that for the period, 1895–1905, it lay outside limits to which the same method had attributed practical certainty.

Am I mistaken in my reasoning above, or are all those really mistakes in the Wikipedia? I have seen similar statements all over the place in textbooks, and I am honestly wondering what I am missing.

  • $\begingroup$ This is a much more concrete version of the question I asked earlier math.stackexchange.com/questions/775788/…, and that I would ask the dear moderators to delete, as it was too vague to be useful. Please forgive me partially reposting something, I hope you will understand making a complicated reasoning clear does not always come easy or quickly. I will not post anything similar again. $\endgroup$ – Jarosław Rzeszótko May 1 '14 at 20:07
  • 1
    $\begingroup$ The law of large numbers is a red herring, I think: you're stuck on the idea of expressing "physical" quantities (such as the result of a frequency-measuring experiment) as random variables. $\endgroup$ – Hurkyl May 1 '14 at 21:18
  • $\begingroup$ You can express a frequency-measuring experiment as X-dash as defined above regardless of what P() is, but the moment you take expectations, and multiply the P()'s of particular values of the random variable by the actual values, you end up with a statement about what we intuitively think as the mean value from repetitions of the experiment only with additional assumptions about P()'s that are not in the axioms of Kolmogorov. That is indeed where my disagreement with Wikipedia and its interpretation of LLN has its roots, but you seem to claim I am simply misunderstanding something here, right? $\endgroup$ – Jarosław Rzeszótko May 1 '14 at 21:53
  • $\begingroup$ better work the example out what is really the variance. $\endgroup$ – Willemien May 2 '14 at 6:43
  • $\begingroup$ @Willemien I am not sure what you mean? $\endgroup$ – Jarosław Rzeszótko May 2 '14 at 6:58

Kolmogorov's axioms, if one were to make an assumption about the distribution of the random variable $X_i$, could be used to derive the distribution of the random variable $\bar{X}$. Notice in the last statement that since $X_i$ is a random variable, $\bar{X}$ is also a random variable. The fact that $\bar{X}$ is a random variable means that there is a probability measure for the random variable $\bar{X}$. The beauty of the WLLN is that so long as both $\mu$ and $\sigma^2$ are finite, no assumptions about the measure $P()$ must be made in order to derive that $\bar{X_n}$ converges in probability to $\mu$. I agree with Hurkyl. Perhaps this post will help with the concept of a random variable https://stats.stackexchange.com/questions/50/what-is-meant-by-a-random-variable

You do make a good point, however, about whether or not the assumptions that the $X$'s are independent and identically distributed random variables may not be true in practice, which is the problem alluded to in the Keynes example.

The example regarding dice appears to rely on the assumption that the die is fair, which may or may not be reasonable depending on how the die is constructed and rolled. However, it seems reasonable to assume that there exists appropriate setups of a dice rolling experiments for which the rolls are $i.i.d$ random variables with a probability measure $P$. In such a case, it does follow from the WLLN that $\bar{X}$ would indeed converge to $\mu$.

  • $\begingroup$ I have no doubts that X-dash converges to mu in the framework of the Kolmogorov's axioms, but the question is whether this allows to draw any interpretable conclusions. Based only on the axioms, mu and X-dash are not interpretable as the average value from a large result of trials, they are simply weighted averages of some set of values using the, arbitrary to some extent, measure P(). Similarly when "relative frequency" is mentioned in the context of the theory, I think it does not really translate into real world relative frequency, unless the WLLN is assumed as true a priori. $\endgroup$ – Jarosław Rzeszótko May 3 '14 at 13:04
  • $\begingroup$ In other words, it seems to me people widely fail to notice that when "relative frequency" is spoken of in the context of probability theory, it only corresponds to our intuitive notion of "relative frequency", if one makes assumptions additional to the Kolmogorov's axioms, and the assumption needed is the WLLN itself. Hence no conclusions about real world situations follow purely from the WLLN as derived from the axioms. $\endgroup$ – Jarosław Rzeszótko May 3 '14 at 13:05
  • $\begingroup$ By the way, the trials in the Keynes examples are independent and identically distributed, the problem is that the probabilities are slightly off from the ideal theoretical relative frequency in an infinite limit of trials. While such P()'s satisfy the axioms, and the formal mathematics stays "true", you see that the result does not seem to be true anymore, and that is because the intuitive interpretation of the various terms in the derivation does not hold anymore. This example shows the WLLN has to be assumed a priori for the usual real world interpretation to hold. $\endgroup$ – Jarosław Rzeszótko May 3 '14 at 13:19
  • $\begingroup$ Based on the axions, $\mu$, the expected value is calculated mathematically and is not based on a relative frequency argument. You are correct the large trial interpretation of $\mu$ that it is the average of a large number of trials would imply that WLLN would be circular, but I would argue that is based solely on that particular interpretation of $\mu$. $\endgroup$ – jsk May 3 '14 at 16:23
  • $\begingroup$ In regards to the relative frequency interpretation of probability, that is again only the frequentist interpretation of probability. There is nothing in Kolmogorov's axioms which states that you should invoke the relative frequency interpretation of probability. $\endgroup$ – jsk May 3 '14 at 16:27

You are correct. The Law of Large Numbers does not actually say as much as we would like to believe. Confusion arises because we try to ascribe too much philosophical importance to it. There is a reason that the Wikipedia article puts quotes around 'guarantees' because nobody actually believes that some formal theory (on its own) guarantees anything about the real world. All LLN says is that some notion of probability, without interpretation, approaches 1 -- nothing more, nothing less. It certainly doesn't prove for a fact that relative frequency approaches some probability (what probability?). The key to understanding this is to note that the LLN, as you pointed out, actually uses the term P() in its own statement. I will use this version of the LLN:

"The probability of a particular sampling's frequency distribution resembling the actual probability distribution (to a degree) as it gets large approaches 1."

Interpreting "probability" in the frequentist sense, it becomes this:

Interpret "actual probability distribution": "Suppose that as we take larger samples, they converge to a particular relative frequency distribution..."

Interpret the statement: "... Now if we were given enough instances of n-numbered samplings, the ratio of those that closely resemble (within $\epsilon$) the original frequency distribution vs. those that don't approaches 1 to 0. That is, the relative frequency of the 'correct' instances converges to 1 as you raise both n and the number of instances."

You can imagine it like a table. Suppose for example that our coin has T-H with 50-50 relative frequency. Each row is a sequence of coin tosses (a sampling), and there are several rows -- you're kind of doing several samples in parallel. Now add more columns, i.e. add more tosses to each sequence, and add more rows, increasing the amount of sequences themselves. As we do so, count the number of rows which have a near 50-50 frequency distribution (within some $\epsilon$) , and divide by the total number of rows. This number should certainly approach 1, according to the theorem.

Now some might find this fact very surprising or insightful, and that's pretty much what's causing the whole confusion in the first place. It shouldn't be surprising, because if you look closely at our frequentist interpretation example, we assumed "Suppose for now that our coin has T-H with 50-50 relative frequency." In other words, we have already assumed that any particular sequence of tossings will, with logical certainty, approach a 50-50 frequency split. So is should not be surprising when we say with logical certainty that a progressively larger proportion of these tossing-sequences will resemble 50-50 splits if we toss more in each, and recruit more tossers? It's almost a rephrasing or the original assumption but at a meta-level (we're talking about samples of samples).

So this certainty about the real world (interpreted LLN) only comes from another, assumed certainty about the real world (interpretation of probability).

First of all, with a frequentist interpretation, it is not the LLN that states that a sample will approach the relative frequency distribution -- it's the frequentist interpretation/definition of $P()$ that says this. It sure is easy to think that, though, if we interpret the whole thing inconsistently -- i.e. if we lazily interpret the outer "probability that ... approaches 1" to mean "... approaches certainty" in LLN but leave the inner statement "relative frequency dist. resembles probability dist." up to (different) interpretation. Then of course you get "relative frequency dist. resembles probability dist. in the limit". It's kind of like if you have a limit of an integral of an integral, but you delete the outer integral and apply the limit to the inner integral.

Interestingly, if you interpret probability as a measure of belief, you might get something that sounds less trivial than the frequentist's version: "The degree of belief in 'any sample reflects actual belief measures in its relative frequencies within $\epsilon$ error' approaches certainty as we choose bigger samples." However this is still different from "Samples, as they get larger, approach actual belief measures in their relative frequencies." As an illustration, imagine if you have two sequences $f_n$ and $p_n$. I am sure you can appreciate the difference between $lim_{n \to \infty} P(|f_n - p_n| < \epsilon) = 1$ and $lim_{n \to \infty} |f_n - p_n| = 0$. The latter implies $lim_{n \to \infty} f_n$ = $lim_{n \to \infty} p_n$ (or $=p$ taking $p_n$ to be a constant for simplicity), whereas this is not true for the former. The latter is a very powerful statement, and probability theory cannot prove it, as you suspected.

In fact, you were on the right track with the "absurd belief" argument. Suppose that probability theory were indeed capable of proving this amazing theorem, that "a sample's relative frequency approaches the probability distribution". However, as you've found, there are several interpretations for probability which conflict with each other. To borrow terminology from mathematical logic: you've essentially found two models of probability theory; one satisfies the statement "the rel. frequency distribution approaches $1/2 : 1/2$", and another satisfies the statement "the rel. frequency distribution approaches $1/\pi : (1-1/\pi)$". So the statement "frequency approaches probability" is neither true nor false: it is independent as either one is consistent with the theory. Thus, Kolmogorov's probability theory is not powerful enough to prove a statement in the form "frequency approaches probability". (Now, if you were to force the issue by saying "probability should equal relative frequency" you've essentially trivialized the issue by baking frequentism into the theory. The only possible model for this probability theory would be frequentism or something isomorphic to it, and the statement becomes obvious.)


What you're missing is that the derivation of the WLLN is allowed to use, not only the Kolmogorov axioms, but also the assumption stated in the theorem: "The $X_1,X_2,\dots,X_n$ are a sequence of $n$ independent and identically distributed random variables with the same finite mean μ, and with variance $σ^2$". So, for example, if we are tossing a fair coin, we know that μ=1/2 (this is what "fair coin" means in probability theory), not $1/\sqrt\pi$. And likewise, in a Bernoulli trial, we are given the actual mean to which the observed probabilities are supposed to converge. And Keynes/Czuber's example isn't a valid application of the LLN because we are not given the actual mean and standard deviation.

So the first two claims in the Wikipedia article are basically correct (except that "will converge to the theoretical probability" should read "will converge in probability to the theoretical probability"; the probability that the observed values do not converge to the theoretical value is 0; but it might happen anyway).

However, the third claim, "According to the law of large numbers, if a large number of six-sided die are rolled, the average of their values (sometimes called the sample mean) is likely to be close to 3.5, with the precision increasing as more dice are rolled." doesn't follow, since we don't know a priori that rolling a six-sided die constitutes a Bernoulli trial. Looking at the context, it seems that the fairness of the die is meant as an ambient assumption, since one of the preceding sentences is "For example, a single roll of a six-sided die produces one of the numbers 1, 2, 3, 4, 5, or 6, each with equal probability."


As i have stated in another question, the (axiomatic) theory of probability is a mathematical framework extrapolating a specific model of physical processes.

One of our professors on stochastic theory and probability used to say: "the (application of the) theory of probability only has meaning for processes/systems which exhibit statistical stability".

It is like applying Group theory or Field theory where it does not apply (this makes the confusion betwen the formulation of probability theory and its application and interpretation to the point).

This ansers both the question and the specific counter examples and in a sense unifies the frequestist and bayesian viewpoints.

(btw Jaynes' exposition is one of my favorites but strongly disagree on the subjectivist view which i can debate quite well, but this is not of the essence here).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.