3
$\begingroup$

Context

I have an irregular time series where data points occur at irregular intervals of time. As a way to observe the behavior of this data over time, I want to use some type of moving average.

The traditional suggestions for moving averages such as simple, weighted, or exponential, are not useful due to the irregular frequency of data points.

Date Value
2022-12-04 10
2022-12-03 10
2022-12-02 10
2022-12-01 10
2020-01-01 1

This is an example of what could happen if I blindly applied a simple moving average. Suppose this example data represents a sequence of 5 data points that could occur within my data.

$$ SMA_5 = \frac{10+10+10+10+1}{5} $$ $$ SMA_5 = 8.2 $$

The simple moving average is dragged down by that last observation of 1, even though the observation occurred two years ago and is of little relevance now. What I want is a moving average that would somehow take into consideration how long it has been from time of evaluation to time of observation. A weighted or exponential moving average, therefore, is not useful since it would assign less weight to the last value for the sake of it being the 5th, and not because it was a long time ago.

My Approach

The solution that I've come up with involves the following steps. I apologize in advance, as I am sure the process is unorthodox.

  1. Computing $DaysAgo$, a new column containing the number of days that has passed since the time of evaluation $$ DaysAgo = CutoffDate - ObservedDate $$ Taking an arbitrary cutoff date of 2022-12-05, my sample data now looks like this.
Date Value Days Ago
2022-12-04 10 1
2022-12-03 10 2
2022-12-02 10 3
2022-12-01 10 4
2020-01-01 1 1069
  1. Computing $Weight$, a new column containing a computed weight for each value $$ Weight = \frac{1}{\alpha*DaysAgo}+\frac{\alpha-1}{\alpha} $$ $$ \alpha \geq 1 $$ The idea behind this formula is to let the weight of each observation model the rational function $y = \frac{1}{x}$, so as to achieve decay over time reflected in the weight. $\alpha$ is there to have a parameter that lets us adjust the strength of the decay over time. The sample data now looks like this.
Date Value Days Ago Weight
2022-12-04 10 1 1.0000
2022-12-03 10 2 0.5000
2022-12-02 10 3 0.3333
2022-12-01 10 4 0.2500
2020-01-01 1 1069 0.0009
  1. Computing $WeightAdj$, a new column containing the adjusted values of $Weight$ $$ WeightAdj = \frac{Weight}{\sum Weight} $$ $$ WeightAdj = \frac{Weight}{1.0000+0.5000+0.3333+0.2500+0.0009} $$

The sample data now looks like this.

Date Value Days Ago Weight WeightAdj
2022-12-04 10 1 1.0000 0.4798
2022-12-03 10 2 0.5000 0.2399
2022-12-02 10 3 0.3333 0.1599
2022-12-01 10 4 0.2500 0.1199
2020-01-01 1 1069 0.0009 0.0004

The weight adjustment allows for the following. $$ \sum WeightAdj = 1 $$

  1. Compute $ValueAdj$, a new column containing the adjusted values of $Value$ $$ ValueAdj = Value * WeightAdj $$

The sample data now looks like this.

Date Value Days Ago Weight WeightAdj ValueAdj
2022-12-04 10 1 1.0000 0.4798 4.7978
2022-12-03 10 2 0.5000 0.2399 2.3989
2022-12-02 10 3 0.3333 0.1599 1.5993
2022-12-01 10 4 0.2500 0.1199 1.1995
2020-01-01 1 1069 0.0009 0.0004 0.0004
  1. Finally, we compute the value of my custom moving average

$$ MA_5 = \sum ValueAdj $$ $$ MA_5 = 9.9960 $$ $$ \alpha = 1 $$ In which, we can see that the old observation barely affects the outcome. If I wanted the time decay to be weaker, however, all I have to do is adjust $\alpha$.

$$ MA_5 = 8.7284 $$ $$ \alpha = 2 $$ $$ MA_5 = 8.2795 $$ $$ \alpha = 10 $$

This seems to me like a novel solution to my problem. It lets me analyze my data over time if I'm willing to compute this moving average many times while shifting the cutoff date by, say, a day each time. Not to mention, this moving average doesn't freak out if you have many points that occurred at the exact same time, or a singular point that occurred a very long time ago.

My Questions

  1. Does this type of moving average exist already?
  2. Does this type of moving average remind you of another one that is more efficient?
  3. Are there any cases where this moving average behaves unexpectedly?

As I got to coding this solution in Python, I figured it would be a good idea to ask about it here first. I welcome any criticisms or anyone who's got a more elegant solution. Your thoughts are much appreciated, thanks.

$\endgroup$

1 Answer 1

1
$\begingroup$

Your calculation is similar to an exponential weighted average. The exponentially-decaying weights are justified by (a sort of vague) analogy to a physical process like the temperature of an object outside. Since the weight function you chose decays slower than exponentially, I'd expect your average in the long run (on a data set with a very large number of old points) to increasingly under-weight new data and favor the old data. Exponential weights avoid this problem.

The EMA (for related reasons) has the advantage that it can be easily updated when a new data point arrives: just combine the old average with the new value using a weight corresponding to the decay factor for the elapsed time.

There's also an efficient algorithm for maintaining a simple moving average with a fixed window duration on an irregular time series. Essentially you maintain a double-ended list of the data points that fall within the window, and you can keep the running average up to date as part of adding new points to the front and removing obsolete points from the back (without re-processing the unaffected internal points).

$\endgroup$
1
  • $\begingroup$ Here is a good source for this adaptation of moving averages: - Andreas Eckner (2017), Algorithms for Unevenly-Spaced Time Series: Moving Averages and Other Rolling Operators. eckner.com/papers/… $\endgroup$
    – hjbello
    Commented Jun 15, 2023 at 12:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .