I have a large set of values $t = \{t_i\}_{i=1}^N$. In actuality, these values (in some set of units) can range between $0$ and an unknown cutoff of the order of $10^7$, but they come from a numerical simulation which, due to memory issues, I have to downsample, so in the course of the simulation I have dropped all $t_i<5.0$.

I would like to calculate the cumulative probability that $t > T$. When I count the number of $t_i$ greater than $T$, and I plot it versus $T$, I get a nice looking truncated power law type distribution for the counts $N(t>T)$ across the variable $T$.

However, I cannot simply write $ P(t>T) = N(t>T)/N$, because I neglected very many values at $T<5.0$, and I should really be normalizing by the total number of my values, including those I neglected, rather than the size of my downsampled data.

That is, the largest value of $P(t>T)$ should happen at $T=0$, and not at $T=5.0$, which is where it would occur if I did it this way.

How can I handle a truncated dataset of this form? I need to calculate a histogram using the frequency of occurrence of values, but I have no means to normalize the counts, because I don't know how many values should actually exist if I hadn't truncated the data.

Any help is appreciated! Thanks

  • $\begingroup$ You actually can estimate $\mathsf{P}(t>T)$ for $T\ge 5$. Just divide by the total number of samples, including the neglected ones. $\endgroup$ – d.k.o. Feb 21 at 23:40
  • $\begingroup$ Do you have a specific power law in mind? $\endgroup$ – JimB Feb 22 at 2:26
  • $\begingroup$ @d.k.o. unfortunately I can't determine the total number of neglected samples. $\endgroup$ – kevinkayaks Feb 22 at 23:12

Why did you choose $5.0$? Clearly you are losing critical information, and the data set you end up with is not a good sample. If you must downsize the sample, don't downsize it by choosing data truncated arbitrarily, but just choose a random sample which is small enough, if possible. This way you can hope to get a representing sample, which you evidently do not have now.

  • $\begingroup$ So you would suggest sampling the simulation as it runs at uniformly random intervals? A priori I don't see an unbiased way to do this either. I arbitrarily chose the sampling rate as $1/5$ as a compromise between memory and fine-grained resolution. I'm mostly interested in the tail of the cdf, so I thought this would be appropriate, but now I'm realizing I can't resolve the tails if I can't define the cdf from the relative frequencies. I agree with you, the sample becomes biased. I can run the simulations again if I knew a better way to sample them as they run $\endgroup$ – kevinkayaks Feb 21 at 20:07
  • $\begingroup$ I believe I could just sample at every $k$th transition, instead of ignoring a certain subset of transitions... does this seem reasonable? It will be a bit difficult to pin down the memory requirements of each simulation condition-- that becomes the new issue. $\endgroup$ – kevinkayaks Feb 22 at 17:15

This question would be better asked at the Cross Validated Stack Exchange site. However...

If you do have samples from a truncated power law distribution (as compared to a censored distribution where you knew how many observations were below 5), then you can certainly estimate the parameter for the non-truncated distribution if you really know that the whole distribution follows the particular power law.

Suppose the truncated distribution has probability density

$$f(x)=\frac{(k-1) x^{-k}}{5^{1-k}}$$

and you have samples $x_1, x_2, \ldots, x_n$. The maximum likelihood estimator of $k$ is

$$\hat{k}=(\overline{\log x}-\log 5 +1)/(\overline{\log x}-\log5)$$

where $\overline{\log x}=\sum_{i=1}^n \log x_i/n$ (i.e., mean of the logs).

Therefore the un-truncated distribution will have density function


for $x\ge 1$ assuming that the lower bound is 1. You mention a lower bound of $0$ but that particular power law density doesn't converge on the interval $(0,\infty)$. So that's why I asked in my comment above if you had a particular (and specific) power law in mind.

An estimate of the standard error of $\hat{k}$ is

$$\sqrt{\frac{(\hat {k}-1)^2}{n}}$$

  • $\begingroup$ Thanks @JimB. The maximum likelihood estimator is instructive. My process is difficult to study analytically, and meant to represent a physical process whose outcome is experimentally uncertain. I suspect a truncated Pareto distribution (can google Aban et al 2006) at least fits the data, but I need a more clever sampling method in order to study it carefully, I think. I'll move to crossvalidated! $\endgroup$ – kevinkayaks Feb 22 at 16:48
  • 1
    $\begingroup$ A quick glance at that paper (Alban et al 2006) seems to indicate only an upper bound truncation is considered when what you describe is a lower bound truncation. Either or both is pretty simple to do. $\endgroup$ – JimB Feb 22 at 16:57
  • $\begingroup$ The upper bound truncation is a natural result of the process, while the lower bound truncation is a relic of my $1/\Delta t$ sampling rate. Thanks ! MLE on a double truncated distribution is an option, but I think, given my uncertainty about the truncated Pareto form on the data, I need a more careful down-sampling method on the full time-series $\endgroup$ – kevinkayaks Feb 22 at 17:07

As far as I understand you need to restrict the number of stored transitions. Instead of dropping observations, you may store $k$'th observation with probability $0<p<1$. That is, let $X_k\sim\text{Bern}(p)$ independent of $t_k$. Then you store $t_k$ if $X_k=1$ and estimate $q_T:=\mathsf{P}(t>T)$ using $$ \hat{q}_T=n^{-1}\sum_{k=1}^n 1\{t_k>T\}\times 1\{n>0\}, $$ where $n$ is the number of stored samples (note that $n\sim \text{Bin}(p,N)$ where $N$ is the (unknown) total number of observations). Assuming that each $t_k$ is an independent copy of $t$, $$ \mathsf{E}\hat{q}_T=\sum_{l=1}^N l^{-1}\sum_{k=1}^l \mathsf{P}(t_k>T)\times \mathsf{P}(n=l)=\mathsf{P}(t>T)\times \mathsf{P}(n>0), $$ which is very close to $\mathsf{P}(t>T)$ when $N$ is large.

On average you will need to store $pN$ observations with standard deviation $\sqrt{Np(1-p)}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.