# How to find probability of a given sample generated by a given distribution

This is a homework problem I have.

Let a random variable $x$ follow a Gaussian distribution with mean $\mu = 10$ and variance $1$.

Which of the following samples of size 4 has the largest probability of being generated from this distribution?

i. 9, 9.1, 10, 11

ii. 1, 2, 0, 11

iii. 9.9, 10.1, 9.8, 10.3

iv. 11, 9, 12, 8

v. 10, 10, 10, 1

My friend argues that iii. is the right answer because $\Pi_i \, p(x_i;\mu=10,\sigma^2 =1)$ would be higher for it. But I don't think that is true. If it were, a sample size of a million with all values equal to 10 would become most likely to be generated.

In my opinion, we need to take sample mean and variance, and find their likelihood from the distribution of sample mean and variance.

• Given the null hypothesis, your friend's answer (iii) has the highest likelihood of the five options, while seeing something like (i) or (iv) would be less suggestive than (iii) of having come from a normal distribution with other parameters (especially one with a smaller variance) – Henry Jan 10 '18 at 18:13
• But we're given that the distribution is fixed, so considering other distributions is a mistake. – David G. Stork Jan 10 '18 at 18:26
• Exactly as posed, the problem is a 'mistake' because the probability of each choice is $0.$ Notice that @Henry answered in terms of likelihoods. – BruceET Jan 11 '18 at 0:03

Find the product of the probability densities of each of the four points in the sample, proportional to:

$$\prod_{i=1}^4 P(x_i| {\cal N}(\mu, \sigma))$$

The values are:

i) $.00621522$

ii) $9.66 \times 10^{-56}$

iii) $0.0235$

iv) $0.000170674$

v) $6.527 \times 10^{-20}$

• <pedant>Those are densities rather than probabilities</pedant> – Henry Jan 10 '18 at 18:14
• Yes they are. The calculation is still correct. – David G. Stork Jan 10 '18 at 18:15
• More seriously, this does not address the apparent paradox that data coming from a random variable with the same mean and smaller variance will typically have a higher value for this product than data actually coming from this distribution – Henry Jan 10 '18 at 18:16
• What's the problem? If I have ${\cal N}(0,1)$ and happen to generate a single point at $x = .3$, then the probability of getting that point would be higher if the normal had instead been ${\cal N}(.3, .000001)$. So what? In this problem the distribution is fixed. – David G. Stork Jan 10 '18 at 18:19
• I downvoted because one should always distinguish probabilities from probability densities. While I understand what the person who wrote the question must have meant, they still shouldn't have used the more popular meaning of the word "probability". – JimB Jan 11 '18 at 1:51

Find $S=\frac14\sum_{i=1}^4(X_i-10)^2$. Select the sample for which the corresponding $S$ is closest to $1$.

The stated 'probability' criterion does not make sense. So one is left to imagine what might have been intended. One approach for parts (i)-(iv) is to do a Kolmogorov-Smirnov goodness-of-fit test of each sample to $\mathsf{Norm}(\mu=10, \sigma=1).$

Roughly speaking this test compares the 'distance' of the empirical CDF of the sample from the CDF of the candidate distribution. This distance has a known distribution when the candidate distribution is correct (the null hypothesis is true).

The P-value of the test is the probability that a sample of the given size would be 'farther' from the candidate distribution then the one tested (under the null distribution). Thus large P-values might be taken as an indication of a relatively good fit and small ones as an indication of a relatively bad fit.

In R statistical software the test for (iv) gives the following output:

x4
##  11  9 12  8

ks.test(x4, pnorm, 10, 1)

One-sample Kolmogorov-Smirnov test

data:  x4
D = 0.3413, p-value = 0.6347
alternative hypothesis: two-sided


Of the four samples tested, (iv) has the largest P-value. P-values for samples (i)-(iii) are 0.1875, 0.0058, and 0.3737, respectively. So, of the four samples tested, sample (iv) has the best fit to $\mathsf{Norm}(10,1)$ according to the K-S criterion.

The K-S test is for continuous distributions, and so does not work with samples such as (v) which have tied values. I believe that sample (iv) can be eliminated because if samples of size 4 are taken from $\mathsf{Norm}(10,1)$ and rounded to two places, the probability of seeing only two unique values is about 0.00006. (And about 0.006, if rounded to one place.) Moreover, the 'unrounded' sample $(9.99, 10.00, 10.01, 1.00)$ has K-S P-value 0.1941.

Note: Another criterion might be to try matching sample means and SDs with $\mu = 10$ and $\sigma = 1.$ Samples (iii) and (iv) are the only plausible choices:

mean(x3); sd(x3)
## 10.025
## 0.2217356
mean(x4); sd(x4)
## 10
## 1.825742


Again here, I'd choose sample (iv) because it has no ties, the smallest difference in means, and the most favorable ratio of SDs. (See @Henry's Comment on variances.)

• I get 0.7200191, 0.0078125, 0.373661, 0.6347476, and 0.2699997 for the respective P-values from the ks.test with (i) getting the largest P-value using R. I'd pick (i) because the "distance" from the sample mean and standard deviation to the mean of 10 and standard deviation of 1 is closest for (i). – JimB Jan 13 '18 at 5:33
• @JimB: You are right about the K-S P-val for (i), so (i) seems better than I said. In my opinion you shouldn't report a K-S P-val for (v) because of the ties; didn't you get a warning message? Thinking about this further in the last couple of days, I concluded this Problem is a fool's errand; the sample sizes are simply too small to make cogent judgments. Especially, with $n = 4,$ taking values of $\bar X$ and $S^2$ as serious estimates of $\mu$ and $\sigma^2$ seems futile. The CIs are quite wide. – BruceET Jan 13 '18 at 16:19
• A similar problem with several models and one (larger) dataset would come closer to what one encounters in actual statistical practice. And then it would be meaningful to look at likelihoods (with or without prior a distribution on the models). – BruceET Jan 13 '18 at 16:23
• No disagreement here. Yes, K-S shouldn't be used when there are ties (not to mention just 4 samples) and I just ignored the warning because given the poorly worded homework/test question, it would be like complaining about the portion size of bad food. – JimB Jan 13 '18 at 18:13