Can Statistics be made Rigorous I am taking a statistics class this term, and this is really my first taste of the subject. We were given the following definition:

Let $f_n(A)$ be the relative frequency of some random event $A$ occurring over $n$ trials. Then if $A$ occurred $0\leq m \leq n$ times: $f_n(A) = \frac{m}{n}$. After repeated trials we will find that each time $f_n(A) \approx p$ for some number $p\in[0,1]$.

We call this the frequentest definition of probability, and really it says that the odds of a random event $A$ occuring is that number $p$. So naturally I think well then we must have:
$$
p = \lim_{n\rightarrow\infty} f_n(A)
$$
However, after more careful thought this cannot be true. Since for a sequence $f_n$ to be convergent we would need for any $\epsilon > 0$ there to exist some $N\in\mathbb{N}$ such that if $n\geq N$ we have: $\left|f_n-p\right| \leq \epsilon$. But since $A$ is a random event we cannot conclude that after so many trials we won't get just a sequence of other events from the sample space that make it look like $p$ is a different value. 
So how do we come up with a rigorous definition of the probability of a random event? Any suggested readings would be greately appreciated.
 A: The Kolmogorov probability axioms may be what you're looking for. These defined probabilities in terms of measure theory, which then allowed for a rigorous treatment of the limit you proposed. In particular, although the sample space includes values outside the bounds of the limit, it can be shown that the probability measure of that space tends to 0. This is called "almost sure" convergence, and is directly analogous to the concept of "almost everywhere" in real analysis.
A: You are right that we need to discuss some other forms of convergence as we are dealing with random events. 
This wikipedia page discusses various forms of convergence of random variable. Notice that $m$ is a binomial distribution with parameter $n$ and $p$. 
By weak law of large number, we can conclude that  it converges in probability, that is 
$$\lim_{n \to \infty}Pr\left(\left| \frac{m}{n} - p\right| > \epsilon \right)=0$$
By strong law of large number, we can conclude that it converges almost surely. that is 
$$Pr\left(\lim_{n \to \infty} \frac{m}{n} =p  \right)=1$$
A: One can easily rigorously prove the result by using the law of large numbers. If $X_1,X_2,\ldots$ is an IID sequence with finite expectation, then almost surely
$$
\lim_{n\rightarrow\infty}\frac{1}{n}\sum_{k=1}^{n}X_k = E(X_1).
$$
In our case we can define $X_k = 1_{A}$, then 
$$
\lim_{n\rightarrow\infty}f_n(A) = \frac{1}{n}\sum_{k=1}^{n}X_k = E(X_1) = P(A).
$$
Your reasoning is right, but you have to realise that as $n$ increases, the probability of an event that significantly moves $f_n$ away from $p$ becomes smaller and smaller. This is because a small number of experiments will have a small impact on the fraction $f_n$, if $n$ is large.
