8
$\begingroup$

I wish to derive a finite (that is, which has a finite support) discrete approximation to a normal distribution, with the following considerations:

  1. It should have exactly the same mean and variance
  2. It must be symmetric
  3. It should resemble the normal distribution in some sense (unimodal pmf, etc.)
  4. It should be discrete and finite (having finite support), with a pre-determined set which contains the support (for example, the integers).

Naive attempt

Here's a naive attempt. Suppose we wish to give an approximation to $\mathcal{N}(\mu,\sigma^2)$. Let the support be $S=\left[\lfloor \mu-d\sigma \rfloor,\lceil \mu+d\sigma\rceil\right]$ (for some natural $d$, perhaps $3$), and define the following pmf: \begin{equation*} f(k) = \begin{cases} \Phi_{\mu,\sigma^2}\left(k+\frac{1}{2}\right) & k = \min{S}\\ \Phi_{\mu,\sigma^2}\left(k+\frac{1}{2}\right) - \Phi_{\mu,\sigma^2}\left(k-\frac{1}{2}\right) & \min{S} < k < \max{S}\\ 1 - \Phi_{\mu,\sigma^2}\left(k-\frac{1}{2}\right) & k = \max{S} \end{cases} \end{equation*} where $\Phi_{\mu,\sigma^2}$ is the cdf of $\mathcal{N}(\mu,\sigma^2)$. This is a legitimate pmf (sums to $1$), it is symmetric, unimodal, discrete and finite, and has mean $\mu$ -- but it does not have variance $\sigma^2$ (I think it always has a larger variance).

Can you fix this naive solution somehow?

$\endgroup$
6
  • $\begingroup$ Do you mean it has finite support? (i.e.$f(k)=0$ for $|k|\geq K$ for some $K>0$) $\endgroup$
    – Dan
    Sep 5, 2014 at 8:59
  • $\begingroup$ Indeed, I'll clarify this. $\endgroup$
    – Bach
    Sep 5, 2014 at 9:00
  • $\begingroup$ If you wish to change the variance for a specified support you will have to change the shape of your distribution away from a Gaussian over that interval. An easy way (but almost certainly not what you're after) is to add two delta functions with suitable coefficients to each end of the support. $\endgroup$
    – Dan
    Sep 5, 2014 at 9:05
  • 3
    $\begingroup$ Your requirements cannot be met for arbitrary mean $\mu$. This is because a symmetric approximating distribution with support $\{ a_1, ... a_n\} $ will have a mean that's necessarily either $a_{(n+1)/2}$ (odd $n$) or $\frac{1}{2}(a_{n/2} + a_{n/2 + 1})$ (even $n$); e.g., if the support is a set of consecutive integers, then the mean must be a multiple of $\frac{1}{2}$. $\endgroup$
    – r.e.s.
    Sep 7, 2014 at 16:59
  • $\begingroup$ (cont'd) In fact, your "naive solution" is not symmetric, and yet does not exactly match an arbitrary mean. For example, with $\mu = 4.9, \sigma = 1$, and support $\{\lfloor \mu- 3 \sigma \rfloor, ..., \lceil \mu + 3 \sigma\rceil\}= \{1,...,8\}$, it gives $f(1)=0.0003369... \ne f(8)=0.004661...$ (not symmetric) and mean $=4.8998... \ne 4.9$ (not an exact match). $\endgroup$
    – r.e.s.
    Sep 7, 2014 at 16:59

3 Answers 3

5
$\begingroup$

Restating the question, what probability mass function with finite support $ P(X=k) $ most excellently approximates the normal distribution $ X \sim \mathcal{N}(\mu, {\sigma}^2) $; mimicking its symmetrical shape, statistical measures, and general behavior?

The Normal Mass Function

$$ P(X=k) = \frac{g(k)}{C} $$

$$ g(k) = e^{\frac{-(k-\mu)^2}{2{\sigma}^2}} $$

$$ C = \sum_{i=1}^{n} g(k_i) \approx \sqrt{2 \pi {\sigma}^2 } $$

The Normal Mass Function (NMF) is a PMF that is exactly proportional to the Normal PDF, and if given an adequate support, is virtually the same function. It performs so well, and shares such a tight connection with its continuous brother, that I had to name it authoritatively. But some limitations must be respected in order to achieve admirable performance, as follows.

The support of the Normal Mass Function must consist of equally spaced points.

$$ \delta = k_{i+1} - k_i $$

I have three free parameters in creating the support $ \{k_1, \ldots, k_n\} $. The parameters are the spacing, the start, and the number of points, namely $ \delta $, $ k_1 $ and $ n $. These parameters may be chosen for one to one comparison with other discrete distributions. However, for best agreement to the normal distribution, the support should not only have equally spaced points, but also be defined symmetrically, with $ \text{median}(k_i) = \mu $, or equivalently:

$$ k_1 = \mu - (n-1) \delta / 2 $$ $$ k_n = \mu + (n-1) \delta / 2 $$

This refined support in context of the same function defines the Symmetric Normal Mass Function.

I have two free parameters in creating the symmetric support: the spacing, and the number of points, namely $ \delta $ and $ n $. If $ (n-1) \delta / \sigma \ge 12 $ and $ \sigma / \delta \ge 1 $ then I have an acceptable support: the Symmetric Normal Mass Function agrees closely with the Normal Distribution. An acceptable support spans at least 6 standard deviations away from the mean (in both directions) and has at least one point per standard deviation.

By these rules, the minimum acceptable support has 13 points spaced at 1 point per standard deviation. Given the minimum acceptable support, the Symmetric Normal Mass Function compared to the Normal Distribution, has

  • identical mean,
  • approximate variance (low, within 2.2 parts per 10 million),
  • identical skewness, and
  • approximate kurtosis (high, within 2.4 parts per million).

$$ \mu, \sigma^2, \frac{\mu_3}{\sigma^3}, \frac{\mu_4}{\sigma^4} \approx \mu, \sigma^2, 0, 3 $$

Except for this minor decimal disagreement, the Symmetric Normal Mass Function meets all 4 requirements of the original question marvelously. The approximation improves as the support's span and density increases relative to the standard deviation.

I expound further about why the Normal Mass Function is the best choice as the discrete variant of the Normal Distribution in a linked article.

$\endgroup$
2
  • $\begingroup$ Excellent answer, thank you! The "Normal Mass Function" link is now broken -- do you have an updated link per chance? Or any more information? It's hard to find anything when I search for "Normal Mass Function". $\endgroup$
    – Brian
    Jan 31, 2021 at 0:16
  • $\begingroup$ Hi @Brian. Yes my website was down for an extended period, but is now back online!!! Thanks for your interest $\endgroup$
    – pvtrs
    Apr 29, 2021 at 12:36
5
$\begingroup$

Just normalize the expected hitting values (# of times process obtains a given integer) of a simple one-dimensional random walk on $\mathbb{Z}$.

This is equivalent to shifting the binomial distribution $B(n,\frac{1}{2})-\frac{n}{2}$, modulo any little odd/even parity issues. Which is why the normal distribution, $N(n,\frac{n}{4})$, is used as an approximation for the binomial distribution.

It is easy to get intuition for why this works by glancing at Pascal's Triangle (the binomial expansion). In other words, if you have say 7 ordered objects which you want to approximate a normal distribution over selecting, just normalize the coefficients of the binomial expansion with exponent $n=7$. Or normalize the seventh row of Pascal's Triangle.

enter image description here

$\endgroup$
1
$\begingroup$

Let me first rephrase your question to make it more concrete to motivate the solution.

The normal distribution is a ubiquitous probability distribution because given a mean and a variance, a normal distribution yields the maximum entropy among all continuous pdfs (see https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution).

Now we seek to find the maximum entropy discrete pdf given a mean, variance, and a range (min, max, step size).

I claim the general solution is a beta-binomial distribution (https://en.wikipedia.org/wiki/Beta-binomial_distribution).

[Proof left as exercise for reader]

Note that ${\displaystyle \mathrm {BetaBin} (n,\alpha ,\beta )}$ has support $x ∈ { 0, …, n }$, mean ${\frac {n\alpha }{\alpha +\beta }}$, and variance ${\frac {n\alpha \beta (\alpha +\beta +n)}{(\alpha +\beta )^{2}(\alpha +\beta +1)}}$

Let's work out an example problem for clarity. Suppose you took a test and your professor says the mean was 75, standard deviation was 15, with a min of 0 and max of 100.

The solution must be of the form ${\displaystyle \mathrm {BetaBin} (100,\alpha ,\beta )}$ to have a support of the integers from 0 to 100. Then $\alpha = 3\beta$ for the mean to be 75. Now $225 = {\frac {300\beta^2 (4\beta +100)}{(4\beta )^{2}(4\beta +1)}}$ yields $\beta = 2$.

Therefore the maximum entropy distribution of test scores is ${\displaystyle \mathrm {BetaBin} (100,6,2)}$plot of pdf

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .