0
$\begingroup$

There is a very nice way to compute the variance of a moving window as detailed by Knuth and Cook and answered locally here, also on a blog here. The method requires you to make use of the data in the window in the form of $(x_N - x_0)$. I can't store the windowed data, but would still like to simulate the behaviour of having a windowed running update. Is there a way to maintain an estimate of the evolving variance using only the previous step parameters $\mu, \sigma^2, d^2$?

Essentially, the parameters give me an estimate for the distribution for the previous time step, then I see a data point, and if it is an "unexpected" point I want the variance to inflate slightly, and if it is "expected" I want the variance to shrink a bit. The mean and variance of my data stream moves around, but eventually converges, and I'd like the running estimate to capture this behaviour.

Here's a contrived example. $\mu$ starts at 0, linearly increases and is then flat at 5 again. $\sigma$ starts at 1, linearly increases and decreases back to 1. The blue line is computed using a window of 30 points. Green line is using the normal streaming equations, but as expected it is computing the params for the whole data sample, not just the window. The orange line is just capping $N = min(N, 30)$ in the normal streaming setting, but it does not adapt quickly enough, or forget the old samples. Any help would be much appreciated! Contrived example

$\endgroup$

2 Answers 2

1
$\begingroup$

I don't think this is possible to do exact. You will need information about the point leaving the window: Consider the two instantiations of moving windows $a = [1, 3, 1, 3, 1, 3]$ and $b = [3, 1, 3, 1, 3, 1]$. Both have a mean of 2, and a variance of 1, but adding a new number (e.g. 2) and removing the oldest number will result in different means over the new windows. However, if you'd solely base your answer on the current mean and standard deviation, they will not differ.

A method you could try to approximate the mean and standard deviation is to do some kind of a running average. Here you would update your mean and standard deviation based on a new number $x$ via: $$\mu := (1-\rho) \mu + \rho x$$

$$\sigma^2 := (1-\rho) \sigma^2 + \rho (x -\mu)^2$$

For a (small) $\rho \in (0,1)$ that can be tuned. This should induce some latency, but when the distribution stays constant for a while, they should eventually converge to the correct numbers.

$\endgroup$
1
  • $\begingroup$ Thank you for this!! I have been fighting with this for a while and this clean and simple solution has been staring at me in the face! xD This is basically an exponentially weighted average right? Do you think there is a more principled way of selecting $\rho$ "on-the-fly" so that it makes use of the distribution, perhaps using the probability of seeing the data point as a weight somehow? I guess that is a bit of chicken-egg scenario because you're using the data to derive the distribution that you're using to estimate the data... $\endgroup$
    – logan
    Commented Feb 22, 2023 at 19:32
0
$\begingroup$

@student13's answer was just what I was looking for. Here are the updated graphs if anyone is interested, I used a $\rho = \frac{1}{15}$: enter image description here

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .