# Methods to estimate a probability distribution from truncated data?

I have a large set of values $$t = \{t_i\}_{i=1}^N$$. In actuality, these values (in some set of units) can range between $$0$$ and an unknown cutoff of the order of $$10^7$$, but they come from a numerical simulation which, due to memory issues, I have to downsample, so in the course of the simulation I have dropped all $$t_i<5.0$$.

I would like to calculate the cumulative probability that $$t > T$$. When I count the number of $$t_i$$ greater than $$T$$, and I plot it versus $$T$$, I get a nice looking truncated power law type distribution for the counts $$N(t>T)$$ across the variable $$T$$.

However, I cannot simply write $$P(t>T) = N(t>T)/N$$, because I neglected very many values at $$T<5.0$$, and I should really be normalizing by the total number of my values, including those I neglected, rather than the size of my downsampled data.

That is, the largest value of $$P(t>T)$$ should happen at $$T=0$$, and not at $$T=5.0$$, which is where it would occur if I did it this way.

How can I handle a truncated dataset of this form? I need to calculate a histogram using the frequency of occurrence of values, but I have no means to normalize the counts, because I don't know how many values should actually exist if I hadn't truncated the data.

Any help is appreciated! Thanks

• You actually can estimate $\mathsf{P}(t>T)$ for $T\ge 5$. Just divide by the total number of samples, including the neglected ones. – d.k.o. Feb 21 at 23:40
• Do you have a specific power law in mind? – JimB Feb 22 at 2:26
• @d.k.o. unfortunately I can't determine the total number of neglected samples. – kevinkayaks Feb 22 at 23:12

Why did you choose $$5.0$$? Clearly you are losing critical information, and the data set you end up with is not a good sample. If you must downsize the sample, don't downsize it by choosing data truncated arbitrarily, but just choose a random sample which is small enough, if possible. This way you can hope to get a representing sample, which you evidently do not have now.

• So you would suggest sampling the simulation as it runs at uniformly random intervals? A priori I don't see an unbiased way to do this either. I arbitrarily chose the sampling rate as $1/5$ as a compromise between memory and fine-grained resolution. I'm mostly interested in the tail of the cdf, so I thought this would be appropriate, but now I'm realizing I can't resolve the tails if I can't define the cdf from the relative frequencies. I agree with you, the sample becomes biased. I can run the simulations again if I knew a better way to sample them as they run – kevinkayaks Feb 21 at 20:07
• I believe I could just sample at every $k$th transition, instead of ignoring a certain subset of transitions... does this seem reasonable? It will be a bit difficult to pin down the memory requirements of each simulation condition-- that becomes the new issue. – kevinkayaks Feb 22 at 17:15

This question would be better asked at the Cross Validated Stack Exchange site. However...

If you do have samples from a truncated power law distribution (as compared to a censored distribution where you knew how many observations were below 5), then you can certainly estimate the parameter for the non-truncated distribution if you really know that the whole distribution follows the particular power law.

Suppose the truncated distribution has probability density

$$f(x)=\frac{(k-1) x^{-k}}{5^{1-k}}$$

and you have samples $$x_1, x_2, \ldots, x_n$$. The maximum likelihood estimator of $$k$$ is

$$\hat{k}=(\overline{\log x}-\log 5 +1)/(\overline{\log x}-\log5)$$

where $$\overline{\log x}=\sum_{i=1}^n \log x_i/n$$ (i.e., mean of the logs).

Therefore the un-truncated distribution will have density function

$$g(x)=(k-1)x^{-k}$$

for $$x\ge 1$$ assuming that the lower bound is 1. You mention a lower bound of $$0$$ but that particular power law density doesn't converge on the interval $$(0,\infty)$$. So that's why I asked in my comment above if you had a particular (and specific) power law in mind.

An estimate of the standard error of $$\hat{k}$$ is

$$\sqrt{\frac{(\hat {k}-1)^2}{n}}$$

• Thanks @JimB. The maximum likelihood estimator is instructive. My process is difficult to study analytically, and meant to represent a physical process whose outcome is experimentally uncertain. I suspect a truncated Pareto distribution (can google Aban et al 2006) at least fits the data, but I need a more clever sampling method in order to study it carefully, I think. I'll move to crossvalidated! – kevinkayaks Feb 22 at 16:48
• A quick glance at that paper (Alban et al 2006) seems to indicate only an upper bound truncation is considered when what you describe is a lower bound truncation. Either or both is pretty simple to do. – JimB Feb 22 at 16:57
• The upper bound truncation is a natural result of the process, while the lower bound truncation is a relic of my $1/\Delta t$ sampling rate. Thanks ! MLE on a double truncated distribution is an option, but I think, given my uncertainty about the truncated Pareto form on the data, I need a more careful down-sampling method on the full time-series – kevinkayaks Feb 22 at 17:07

As far as I understand you need to restrict the number of stored transitions. Instead of dropping observations, you may store $$k$$'th observation with probability $$0. That is, let $$X_k\sim\text{Bern}(p)$$ independent of $$t_k$$. Then you store $$t_k$$ if $$X_k=1$$ and estimate $$q_T:=\mathsf{P}(t>T)$$ using $$\hat{q}_T=n^{-1}\sum_{k=1}^n 1\{t_k>T\}\times 1\{n>0\},$$ where $$n$$ is the number of stored samples (note that $$n\sim \text{Bin}(p,N)$$ where $$N$$ is the (unknown) total number of observations). Assuming that each $$t_k$$ is an independent copy of $$t$$, $$\mathsf{E}\hat{q}_T=\sum_{l=1}^N l^{-1}\sum_{k=1}^l \mathsf{P}(t_k>T)\times \mathsf{P}(n=l)=\mathsf{P}(t>T)\times \mathsf{P}(n>0),$$ which is very close to $$\mathsf{P}(t>T)$$ when $$N$$ is large.

On average you will need to store $$pN$$ observations with standard deviation $$\sqrt{Np(1-p)}$$.