Page 68 in pattern recognition and machine learning (free) says

there are infinitely many probability distributions that could have given rise to the observed finite data set. Indeed, any distribution p(x) that is nonzero at each of the data points x1, . . . , xN is a potential candidate.

Consider flipping a coin, with param $\mu$.

There are infinite number of probability distributions, such as $\mu$ = 0.1, 0.001, 0.2, ....

could give rise to the observed finite data set {H,T}

what does the second part ("any distribution p(x) that is nonzero at each of the data points x1, . . . , xN is a potential candidate") mean?

are "each of the data points" referring to the set {H,T}, or a single element of the set, such as "H"?

I assume "each of the data points" refer to a single element of the set, "any distribution at each of the data points" implies H and T may have different distributions, each element in set has its own distribution, is my understanding right? if it is, it is NOT i.i.d, right?


In statistics and machine learning your primary input is a sample $x_1, ..., x_n$.

Where the sample comes from changes from problem to problem, but in the usual case we assume that each element of the sample comes from the same distribution, sampled independently.

That is, each $x_i$ is sampled from a random variable $X_i$ such that all the $X_i$ are iid, and distributed according to a probability mass function (assuming that the outcomes are discrete) $p$.

The central problem of statistics is, given a sample, figuring out what $p$ is.

The sentence above highlights that by mere deduction we cannot deduce much about the probability distribution the sample originates from.

Indeed, let our sample be $S = {1,2,1,4,5,3}$ for example.

Then we know that $p(3)>0$ (since at least one instance of the sample $3$ was observed). But any $p$ such that $p(n)>0$ for all $n\in S$ will be a possible candidate for $p$.

  • $\begingroup$ thanks for your answer. in this case, S = {1,2,1,4,5,3}, there are 6 data points, each of which may have its distinct distribution, so there are 6 distinct distributions in total, right? $\endgroup$ – JJJohn Jan 8 '20 at 3:01
  • $\begingroup$ @JJJohn usually we assume that the sampled data points are independently sampled from the same distribution $p$. So there is a single distribution, but there are uncountably many candidates for the shape of that distribution $p$. $\endgroup$ – Jsevillamol Jan 8 '20 at 10:49
  • $\begingroup$ Having said that, it could be the case that you have six distributions - imagine for example drawing cards and noting their number without replacement. But that case is significantly harder to analyze (even if its in a sense more realistic for many cases), so its not the usual object of study in statistics. $\endgroup$ – Jsevillamol Jan 8 '20 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.