Let $X$ be a continuous random variable uniform distributed in the interval $[0,1)$. Let $S$ be a set of $10^4$ samples drawn from $X$ and let $S'=\{0.5,\ldots,0.5\}$ be a set of $10^4$ numbers that are all $0.5$. The probability that either sample is drawn is equally likely, i.e. $P(S')=P(S)$, because the samples are drawn independently.
Yet $S$ is somehow more representative of the underlying flat distribution than $S'$ as illustrated in the histograms below (the red dashed line represents the expected frequency).
How can I quantify how representative a sample is of a distribution? Ideally, I would like to obtain a probability $p$ that indicates how likely it is that a given sample $S$ was generated from a given distribution.
This reminded me of the concept of microstates and macrostates in statistical mechanics where a different number of microstates is associated with any particular macrostate.
