Which law in probability theory states the following?

If we have a large enough number of samples, their histogram function converges their true probability density function. (for a continuous random variable)

I know that "In probability theory, the law of large numbers (LLN) is a theorem that describes the result of performing the same experiment a large number of times. According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed."

But this law is just about expected value. It does not include variance or probability density function.

  • $\begingroup$ I don't think there's a law so much as some rules of thumb. First off, you need to assume at least the form of distribution and just estimate the parameters. Otherwise, the longer and thinner the tails, the more data you would need. So I don't think there's a general result that applies to arbitrary distributions. $\endgroup$ – Gregory Grant Jun 16 '15 at 15:37
  • $\begingroup$ thanks for your reply, I have a question regarding their standard deviation: can we use law of large number to say that with large enough samples, their sample standard deviation converges their standard deviation? $\endgroup$ – Julie Jun 16 '15 at 15:41
  • $\begingroup$ You can use convergence properties of estimates of the standard deviation, but they depend on assuming at least the form of the distribution. Though I guess you could use Chebychev's inequality. Are you familiar with Chebychev? $\endgroup$ – Gregory Grant Jun 16 '15 at 15:56
  • $\begingroup$ Thanks I just read about it in Wikipedia, it says "under Chebyshev's inequality a minimum of just 75% of values must lie within two standard deviations of the mean and 89% within three standard deviations" how can this rule say that from large samples we can estimate standard deviation? $\endgroup$ – Julie Jun 16 '15 at 16:01
  • $\begingroup$ Chebyshev won't help, alas: it assumes that your distribution has a finite variance, which isn't one of the hypotheses you gave. As Wikipedia says, "Let $X$ (integrable) be a random variable with finite expected value $\mu$ and finite non-zero variance $\sigma^2$... " [emphasis mine] $\endgroup$ – John Hughes Jun 16 '15 at 16:54

I worry that the statement you've made doesn't quite make sense. Exactly what is the definition of "the probability density function of the samples"? For instance, if you take the uniform probability on the unit interval, after $k$ samples, you'll might say that that "prob. density of the samples" is a function that's $1/k$ at each of the $k$ sample points...but such a function won't converge to the everywhere-one function, for the limit of these individual functions can be nonzero on at most a countable number of points.

As for your followup question about standard deviation, I believe that the answer is "no," for there are distributions whose standard deviation is infinite, but the sample-standard-deviation will always be finite, hence not "close" to the SD of the underlying distribution.

  • $\begingroup$ Thanks, I edited the question. suppose that the random variable is a continuous random variable not discrete. $\endgroup$ – Julie Jun 16 '15 at 16:02
  • $\begingroup$ Right ... that's exactly the case where things don't make sense: with a finite collection of samples, your "sample distribution" will be discrete, and hence won't converge to the distribution of the RV. If you mean "the distribution is from a known, few-parameter family like Gaussians, and my "sample dist." is the best-fit Gaussian for the samples," that's an altogether different question. Once again: What definition are you using for "the probability density function of the samples"? Only once we know that can we hope to answer your question carefully. $\endgroup$ – John Hughes Jun 16 '15 at 16:15
  • $\begingroup$ Thank you, yes you are right many parameters can affect the answer. Suppose that we do not know the distribution form and we have a large number of samples, using which we want to estimate the PDF of the underlying continuous random variable. $\endgroup$ – Julie Jun 16 '15 at 16:23
  • $\begingroup$ In that case, my answer stands: your statement doesn't make sense until you define what you mean by "the probability density function of the samples". Sorry I can't fix this, but I really don't know what it means. $\endgroup$ – John Hughes Jun 16 '15 at 16:32
  • $\begingroup$ thanks, I also found this en.wikipedia.org/wiki/… $\endgroup$ – Julie Jun 16 '15 at 21:17

Probably the result closest to what you're saying would the Glivenko-Cantelli theorem: https://en.wikipedia.org/wiki/Glivenko%E2%80%93Cantelli_theorem. This states that the empirical distribution function of a random sample converges in a certain sense to the true distribution function as the sample size tends to infinity.

  • $\begingroup$ thank a lot. what is the relationship between the Glivenko-Cantelli theorem and this en.wikipedia.org/wiki/…? $\endgroup$ – Julie Jun 17 '15 at 19:05
  • $\begingroup$ The first statement there refers to pointwise convergence of the empirical distribution function, which is to say the proportion of values not exceeding any t converges to the true probability. This is nothing but the law of large numbers, so the interesting statement is the second which says that the distance between the true and observed functions (under the supremum norm definition of distance) itself goes to zero. The second statement is exactly the Glivenko-Cantelli theorem. $\endgroup$ – dsaxton Jun 17 '15 at 20:33

Your statement is rather imprecise. One could assume the following (but perhaps this is restrictive) : the density $f(x)$ has support on $[a,b]$. Let $g_{n,m}(x)$ be the histogram constructed by taking $n$ samples and dividing the support interval in $m$ segments of same length $h=(b-a)/m$. Let $x_i$ be the center point of each histogram segment.

Then $g_{n,m}(x_i)$ is a Binomial $B(n,p)$ variable with $p=\int_{x_i-h/2}^{x_i+h/2} f(x) dx = I_{x_i,h}\approx h f(x_i)$

This approximation holds if $h \to 0$ and the function has bounded derivative.

Let $$w_{n,m}(x_i)=\frac{g_{n,m}(x_i)}{nh}$$ be the normalized histogram. Then $$E\left(w_{n,m}(x_i)\right)= \frac{ I_{x_i,h}}{h}\approx f(x_i)$$

$$Var\left(w_{n,m}(x_i)\right)= \frac{1}{n h^2} I_{x_i,h}(1-I_{x_i,h})\approx \frac{ f(x_i)}{n h}$$

Then, if the above condition holds, and $h\to 0$, and $n h \to \infty$, the histogram is asymptotically unbiased, and it's variance tends to zero, hence it converges in mean square (and hence in probability).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.