0
$\begingroup$

I want to know of a way to estimate the percentile of potentially highly skewed data where I will not have access to all of the data at once. We can assume that in general, despite the fact that the data may not fit a normal bell curve, it will still be unimodal. In particular, all I have access to is information I was able to tally as I encountered each data point, such as the minimum and maximum values encountered so far, # of values encountered, the sum of all values, the sum of the squares of each value, and the sum of the cubes for each value (for computing the skew), and the sum of the 4th power of each value (for computing the kurtosis). In theory, I could track other information as I obtain each data point, but for now, this is what I have.

Obviously a very cheap estimate would be to just assume that the distribution was normal, and it is very simple to calculate what data points correspond to a given percentile figure, but this estimate does not account for how skewed the data might be, nor for how narrow its modal peak may be.

So given only the above information, is it possible to establish an estimate of what some percentile given values should be that properly accounts for how the data actually seems to be distributed based on the above tallies?

$\endgroup$
0
$\begingroup$

A solution would be to compute the inverse cumulative distribution function of a Pearson system (alternatively to a Johnson system) with your estimation of avg, std, kurtosis and skewness.

In matlab you can use this function to compute the pdf https://fr.mathworks.com/matlabcentral/fileexchange/26516-pearspdf you could cumsum it to have cdf which is increasing so it can be easily inverted.

Edit : Alternatively, you can use R and use the qpearson function https://cran.r-project.org/web/packages/PearsonDS/PearsonDS.pdf which would directly answer your problem.

$\endgroup$
  • $\begingroup$ Thank you, but regretfully, I cannot say that I understand the solution you are proposing. I expect if I did, I might know how to apply it to solving my problem, but I'm afraid that's outside of my current domain of understanding. I would also want to know how to find the answer without relying on any third party math software. $\endgroup$ – Mark Jul 22 '17 at 21:55
  • $\begingroup$ The expression is very complicated to write manually. If you prefer you can use R which is freeware and use the qpearson function cran.r-project.org/web/packages/PearsonDS/PearsonDS.pdf which would directly answer your problem without needing to invert or cumsum a function. $\endgroup$ – Ninjasam Jul 22 '17 at 22:59
  • $\begingroup$ How complex the expression is should not be a problem... a computer will still be doing the work. Again, however... for my purposes, it is not feasible to rely on third party math software. I do not need to write an entire math software system, I only need what I described, above. $\endgroup$ – Mark Jul 24 '17 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.