My definition of empirical probability:

Let $(X_1,\ldots,X_n)$ be a random sample of i.i.d random variables. Then, the empirical probability for a subset A contained in the sample space $\Omega$ is $P_n(A)=\frac{1}{n}\sum_{i=1}^n \mathbb{1}(X_i \in A).$ Fine.

What I don't get is that $P_n(A)$ is a random variable for a fixed event A. Therefore, won't it have it's own (theoretical) probability distribution?

My understanding of empirical probability with an example:

Say you roll a 6 sided die and you introduce the random variable X, where X=1 if you roll a $1$, $X=2$ if you roll a $2$ etc.

You don't know anything about how $X$ is distributed (it may or may not be a fair die for example).

Roll the same die $n$ times, producing the random sample $(X_1,\ldots,X_n)$

Let $A=\{1\}$ and $P(X \in A)=p$. By defintion, $P_n(A) = \{\text{counting the number of times 1 appears}\}/n$. Since the indicator function in this case is a random variable that has a Bernoulli distribution with parameter $p$, $nP_n(A)$ is just a random variable that has a binomial distribution with parameters $p$ and $n$.

Why then is the empirical probability called "empirical" as if to suggest its distribution would be generated by experiment? I kind of just view this as related to (if not a generalization of) the sampling distribution of sample proportions, where $X_i$ are not necessarily Bernoulli.


$P_n(A)$ is a random variable and it does have its own probability distribution, if you know (or impose) a distribution on the sample. That is precisely what you did when you set $\Pr[X \in A] = p$. But the reason why $P_n(A)$ is empirical is because, a priori, there is no assumption on the sampling distribution: the calculation of the statistic $P_n(A)$ does not depend on any parametric assumptions. You don't necessarily know or assume anything about the model from which the observations in your sample were generated.

  • $\begingroup$ Thanks for the response. I see what you're saying, but I haven't defined what p is (I've only assumed that it's non-zero). Can we still not make the claim that $P_n(A)$ is binomially distributed, just centered in different places? $\endgroup$ – Winston Jul 21 '16 at 19:15
  • $\begingroup$ @Winston Unfortunately, no. You're implicitly assuming that the random variable $$\mathbb 1(X_i \in A) \sim \operatorname{Bernoulli}(p),$$ and furthermore, that these are IID. Although a wide variety of models could be chosen for $X_i$ itself, the point is that $p$ here is a parameter, thus a distributional assumption is being made. $\endgroup$ – heropup Jul 21 '16 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.