# Estimate the true mean: given only measurements above a known threshold

I have a computer-generated logfile which shows events which took longer than a particular time threshold (say 1 second for the sake of argument): if I measure the mean of these measurements, it seems this mean value will be 'off' because of the unreported measurements that didn't hit the threshold value.

Is there a standard way of re-adjusting the mean to take into account these missing values that failed to meet the threshold : and what assumptions need to be made in order to carry this out ?

-
You'll need some assumptions about the underlying distribution to make a good guess about the correct mean. With computer-based events, this may be hard to do. For an everyday example you may have ping times of a few milleseconds between tow hosts except in very rare cases (once in a ten pings? once in 1000 pings? - makes a large difference for the mean) some router has a hiccup and adds a one second delay. Knowing only the average of the few exceptional problem cases helps little for the all-is-fine scenario. – Hagen von Eitzen Sep 22 '13 at 15:27
So something like : find the shape of the distribution; maybe chucking out the outliers (from your ping example). If the distribution looks like a 'warped' (long tailed?) Normal (or something else), perhaps try and 're-shape' it so that it fits ? I am already making an assumption that the measurements are representative actually (ie, no 'hiccups') - was wondering if there was an 'idealized' way of doing this here.... – monojohnny Sep 23 '13 at 9:34

This is a standard case of truncated data (not to be confused with censored data). So you need to consider density and distribution functions. Since your variable, call it $Y$, reflects time intervals, it will be non-negative. Denote the threshold level above which we get data as $t^*$. Call $Y^*$ the truncated random variable which ranges in $[t^*, \infty]$. Then it is standard that

$$f_{Y^*}(y^*) = \frac {f_Y(y)}{F_Y(y\ge t^*)}= \frac {f_Y(y)}{1-F_Y(y\le t^*)}$$

where $f()$ is a pdf and $F()$ is a cdf.

Then $$E(Y^*) = \int_{t^*}^{\infty} \frac {yf_Y(y)}{1-F_Y(y\le t^*)}dy$$

while

$$E(Y) = \int_{0}^{\infty} yf_Y(y)dy = \int_{0}^{t^*} yf_Y(y)dy+\int_{t^*}^{\infty} yf_Y(y)dy$$

Combining we obtain

$$E(Y) = \int_{0}^{t^*} yf_Y(y)dy+E(Y^*)\left[1-F_Y(y\le t^*)\right]$$

So you need to make assumptions for the distribution that your untruncated variable follows. If you know how many incidents are not reported then you can approximate $F_Y(y\le t^*)$ by the ratio "Unreported / Total". This step will also simplify your life regarding the estimation of $f_Y(y)$: even if you assume a distribution for it, sensibly you will only assume the family, not the exact parametrized density.

The likelihood function of a sample of $m$ incidents, assuming independence and identical distribution, will be

$$L = \left(f_Y(y)\right)^m\left[1-F_Y(y\le t^*)\right]^{-m}$$

and the log-likelihood

$$\ln L = m\ln f_Y(y)-m\left[1-F_Y(y\le t^*)\right]$$

If $F_Y(y\le t^*)$ is estimated as mentioned above, it becomes a constant and you have only to estimate the parameters of the distribution by using the assumed density functional form (such a tactic may be helpful if the cdf is not closed-form and/or difficult to estimate, so it will stress the estimation algorithm).

Once you obtain estimates of the distribution parameters, you have an estimate of the density with values for its parameters - and then you can go back to the expression for $E(Y)$ and obtain an estimation of the untruncated mean. $E(Y^*)$ can be estimated directly by the available data as the truncated sample mean, but you can also obtain an alternative calculation for it using the estimated density, and the expression for $E(Y^*)$ above: comparing these two estimates for $E(Y^*)$ will give you some idea on how reliable your estimation results are.

-
So I am in the embarrassing situation of not being able to tell whether to accept this answer or not - because (I am not proud to say) my maths notation is not good enough to understand the answer fully ! I have upvoted the answer, but I am unable to accept as I cannot verify myself ! – monojohnny Feb 18 '14 at 12:13
If the answer is not useful to you, most certainly you should not accept it. Thanks for the upvote, which, too, is questionable to the degree that you cannot fully follow the answer, as you write. – Alecos Papadopoulos Feb 18 '14 at 12:48
I can upvote it with a clear conscious: I can follow your explanation steps well enough (in general terms) - it's just that it will take me some time to go through in detail and in particular understand how to translate the maths notation into computer code. Additionally: my upvote means "Thanks for the pointers - I can now go and research (for instance 'truncated data' and 'cdf' etc.)" – monojohnny Feb 18 '14 at 14:47