1
$\begingroup$

Assume there is a Set of data which follows a known distribution (e.g. normal distribution).

$$S = \left\{ a_0,a_1 ... a_n \right\}$$

When taking a subset from S

$$S_k = \left\{ a_0,a_1 ... a_k \right\}, S_k \subset S$$

and using it to estimate the parameters of said distribution we end up having the estimated probability density function:

$$\mathcal{N}_e(\hat{\mu}, \hat{\sigma}^2)$$

Is there a way to improve this estimation by adding another point of data to the subset like:

$$S_k^* = S_k \cup \left\{ a_{k+1} \right\}$$

which would yield better parameters for $\mathcal{N}_e$ without recalculating the whole estimation (inspecting every other unit of data again)?

Please note: I have used the normal distribution in this question as example, so a general answer which I could use for other distributions aswell would be even more helpful. Not looking for a detailed step by step solution, if you can point me to a paper or textbook that would be great aswell!

$\endgroup$

1 Answer 1

2
$\begingroup$

Sometimes $Yes$; sometimes $No.$ It depends on the distributional family from which the data are sampled.

It helps to think in terms of 'sufficient statistics' for estimating parameter values of a particular distribution. Also, in what follows 'appropriate' estimator is shorthand for the 'maximum likelihood estimator'.

For example, suppose we are sampling from $N(\mu, \sigma_0^2),$ where $\mu$ is unknown and to be estimated, but $\sigma_0^2$ is a known positive constant. Then for a random sample $X_1, X_2, \dots, X_n$ from this distribution, a sufficient statistic is $T_n = \sum_{i=1}^n X_i,$ and the appropriate estimate of the unknown population mean $\mu$ is $\hat \mu_n = \bar X_n = T_n/n.$

Now if you acquire an additional randomly chosen value $X_{n+1},$ then it is clear how to 'update' $\hat \mu_n$ to $\hat \mu_{n+1}$ without using (or knowing) the individual values $X_1, X_2, \dots, X_n:$ Simply update the sufficient statistic to get $T_{n+1} = T_n + X_n,$ and the rest follows.

Similarly, if both $\mu$ and $\sigma^2$ are to be estimated, the sufficient statistics for a random sample of size $n$ are $T_n$ (as above) and $Q_n = \sum_{i=1}^n X_i^2.$ The appropriate estimate is $$\hat \sigma_n^2 = S_n^2 = \frac{1}{n-1}[Q_n - nT_n^2].$$ Again here, it is a simple matter to update to $\hat \mu_{n+1}$ and $\hat \sigma_{n+1}^2$ given an additional random observation $X_{n+1}.$

However, updating is not always so easy. As a simple example, suppose you have a sample of size $n$ from a Laplace (sometimes called 'double exponential') distribution with unknown center $\mu$ and known scale. (Google 'Laplace distribution', perhaps starting with the Wikipedia article.)

Here the sufficient statistic is the sample median, and the median itself if the estimate of the center (both mean and median) of the distribution. But there is no simple formula for updating the sample median $H_n$ to $H_{n+1}$ given a new observation $X_{n+1}.$ (Similarly, if the scale parameter also needs to be estimated, there is no simple formula for updating its appropriate estimator.)

Roughly speaking then, the ease of updating maximum likelihood estimators given additional observations is determined by the ease of updating the sufficient statistics.

Note: (a) There are methods of estimation other than maximum likelihood, and some of them do not use sufficient statistics. If the estimator is a sum, product, maximum, or minimum of the data, then updating can be easy. More complex formulas and procedures that require completely re-sorting the data may not be easily updatable. (b) It may be worth your while to consult the chapter on estimation in a mathematical statistics text for more information on different kinds of estimation. (c) I have purposely used different symbols in my Answers than the ones in your Question. I believe my notation is closer to what you will find in references.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .