How do I figure out what kind of distribution this is?

Question

I've sampled a real world process, network ping times. The "round-trip-time" is measured in milliseconds. Results are plotted in a histogram:

Ping times have a minimum value, but a long upper tail.

I want to know what statistical distribution this is, and how to estimate its parameters.

Even though the distribution is not a normal distribution, I can still show what I am trying to achieve.

The normal distribution uses the function:

with the two parameters

• μ (mean)
• σ² (variance)

Parameter estimation

The formulas for estimating the two parameters are:

Applying these formulas against the data I have in Excel, I get:

• μ = 10.9558 (mean)
• σ² = 67.4578 (variance)

With these parameters I can plot the "normal" distribution over top my sampled data:

Obviously it's not a normal distribution. A normal distribution has an infinite top and bottom tail, and is symmetrical. This distribution is not symmetrical.

---

What principles would I apply, what flowchart, would I apply to determine what kind of distribution this is?

And cutting to the chase, what is the formula for that distribution, and what are the formulas to estimate its parameters?

---

I want to get the distribution so I can get the "average" value, as well as the "spread":

I am actually plotting the histrogram in software, and I want to overlay the theoretical distribution:

Tags: sampling, statistics, parameter-estimation, normal-distribution

Answer 1

I'd vote for a Poisson distribution with a constant offset.

A hand-wavy reasoning might be that the round trip time is due to a constant offset being the best-case round-trip time assuming no delays due to router queues (= wave propagation velocity over physical distance, + minimum processing time), with "rare events" (see wikipedia page) corresponding to queueing delays in one or more routers that make up the network path(s).

As far as parameter estimation goes, I'm not familiar with how to do it for samples taken from a (suspected) Poisson distribution, but I'm sure you could find something on the Internet.

aha, here we go: http://en.wikipedia.org/wiki/Poisson_distribution#Parameter_estimation -- you could use this after subtracting off the minimum of a large number of samples.

---

drat, stupid me, I glossed over the fact that Poisson = discrete probability distribution.

Answer 2

Sounds an awful lot like the conditions you would expect for an Erlang distribution to me- it also looks a lot like one...

Erlang distributions model the times between occurrences in poisson processes and are frequently used as parts of models of internet traffic.

My interpretation is this: as a site sending back a signal, one processes and sends stuff for a given user in an approximately poisson process (the approximate 'limit' of bernoilli trials p-> stuff for user 1-p -> stuff for a different user) and the time spent waiting for one to occur is therefore distributed Erlangwise, with a shift to the right (to account for the user's sending of the signal). This gives the shape you have above :)

Edit: This should be Erlang-2 if that is not already clear, since receiving and sending are two poisson occurrences from the same distribution depending (as indicated above) on traffic [That is: occurrence 1- server has free bit to process receiving, occurrence 2- computer has free bit to process sending]

Answer 3

The go-to distribution for things like wait times is the Exponential. Yours doesn't look exactly the same because of the tiny lower tail, but I would be inclined to attribute that to noise/measurement error. (The assumption of independence of events is almost certainly wrong for ping times, but it's probably still your best choice.)

Also, you would probably be better off asking this sort of question on the stats site.

---

Edit: As pointed out by Srikant Vadali, the Gamma distribution is more general and can account for a non-negligible short tail, so may be a more appropriate choice. It's easier to estimate the parameter for the exponential, though.

Answer 4

From the comments on stats.stackexchange, it seems like you may not care too much about the distribution, but just a pretty curve to overlay on your graph. In which case, some kind of spline is your best bet. Use some kind of curves with asymptotes at y=0 for your upper- and lower-most segments, and whatever fits best in between.

If you do actually care about the underlying distribution:
The first step would be to use whatever outside knowledge you have to characterize the distribution. For example:
Network ping is a sum of independent wait times (the individual nodes in the network). This would suggest a Gamma/Erlang distribution if each of these steps is identical, and a more complex distribution if they are not.
Ping is a measure of time until the computer at the other end responds to your request, the likelihood of which is proportional to the time elapsed. This would suggest a Weibull distribution.
Ping time is the accumulation of a large number of factors that all have a multiplicative effect on the result. Then a log-normal distribution would be best. I don't know enough about networking to say anything about the accuracy of any of the above models, and it's also perfectly likely that ping time follows some other model which I haven't thought of. I just wanted to demonstrate the idea: that you should think about what factors contribute to the thing you are trying to model, and how they interact.

And, of course, the distribution does not necessarily have to be a known one! In which case the above won't get you very far! In this case you might want to come up with your own empirical distribution, for which a variety of methods exist. The most common are to take your measurements as the distribution (as long as you have a sufficiently large number) or to take each of those data points and treat it as the center of some uniform/normal/other distribution, and sum everything with appropriate scaling.

After you know the type of distribution, you may also be able to use domain knowledge to estimate some of its parameters. For example, you might guess at the number of exponentials being summed based on the shape of the network. You can also use your measured mean and variance to form estimates of the distribution parameters. For example, if you thought that your distribution was a Gamma(3,θ), then you could use your measured variance to estimate θ=4.74182454 based on our known formula for variance of a Gamma Distribution.

Once you have your guess at a distribution, you will want to test its goodness of fit.

For this, the standard method would be to apply the one-sample Kolmogorov-Smirnov test.

Other potentially applicable tests are the Cramer-von-Mises, Anderson-Darling, or chi-square tests.

This is incomplete, I will add more later.