Context
I have an irregular time series where data points occur at irregular intervals of time. As a way to observe the behavior of this data over time, I want to use some type of moving average.
The traditional suggestions for moving averages such as simple, weighted, or exponential, are not useful due to the irregular frequency of data points.
Date | Value |
---|---|
2022-12-04 | 10 |
2022-12-03 | 10 |
2022-12-02 | 10 |
2022-12-01 | 10 |
2020-01-01 | 1 |
This is an example of what could happen if I blindly applied a simple moving average. Suppose this example data represents a sequence of 5 data points that could occur within my data.
$$ SMA_5 = \frac{10+10+10+10+1}{5} $$ $$ SMA_5 = 8.2 $$
The simple moving average is dragged down by that last observation of 1, even though the observation occurred two years ago and is of little relevance now. What I want is a moving average that would somehow take into consideration how long it has been from time of evaluation to time of observation. A weighted or exponential moving average, therefore, is not useful since it would assign less weight to the last value for the sake of it being the 5th, and not because it was a long time ago.
My Approach
The solution that I've come up with involves the following steps. I apologize in advance, as I am sure the process is unorthodox.
- Computing $DaysAgo$, a new column containing the number of days that has passed since the time of evaluation $$ DaysAgo = CutoffDate - ObservedDate $$ Taking an arbitrary cutoff date of 2022-12-05, my sample data now looks like this.
Date | Value | Days Ago |
---|---|---|
2022-12-04 | 10 | 1 |
2022-12-03 | 10 | 2 |
2022-12-02 | 10 | 3 |
2022-12-01 | 10 | 4 |
2020-01-01 | 1 | 1069 |
- Computing $Weight$, a new column containing a computed weight for each value $$ Weight = \frac{1}{\alpha*DaysAgo}+\frac{\alpha-1}{\alpha} $$ $$ \alpha \geq 1 $$ The idea behind this formula is to let the weight of each observation model the rational function $y = \frac{1}{x}$, so as to achieve decay over time reflected in the weight. $\alpha$ is there to have a parameter that lets us adjust the strength of the decay over time. The sample data now looks like this.
Date | Value | Days Ago | Weight |
---|---|---|---|
2022-12-04 | 10 | 1 | 1.0000 |
2022-12-03 | 10 | 2 | 0.5000 |
2022-12-02 | 10 | 3 | 0.3333 |
2022-12-01 | 10 | 4 | 0.2500 |
2020-01-01 | 1 | 1069 | 0.0009 |
- Computing $WeightAdj$, a new column containing the adjusted values of $Weight$ $$ WeightAdj = \frac{Weight}{\sum Weight} $$ $$ WeightAdj = \frac{Weight}{1.0000+0.5000+0.3333+0.2500+0.0009} $$
The sample data now looks like this.
Date | Value | Days Ago | Weight | WeightAdj |
---|---|---|---|---|
2022-12-04 | 10 | 1 | 1.0000 | 0.4798 |
2022-12-03 | 10 | 2 | 0.5000 | 0.2399 |
2022-12-02 | 10 | 3 | 0.3333 | 0.1599 |
2022-12-01 | 10 | 4 | 0.2500 | 0.1199 |
2020-01-01 | 1 | 1069 | 0.0009 | 0.0004 |
The weight adjustment allows for the following. $$ \sum WeightAdj = 1 $$
- Compute $ValueAdj$, a new column containing the adjusted values of $Value$ $$ ValueAdj = Value * WeightAdj $$
The sample data now looks like this.
Date | Value | Days Ago | Weight | WeightAdj | ValueAdj |
---|---|---|---|---|---|
2022-12-04 | 10 | 1 | 1.0000 | 0.4798 | 4.7978 |
2022-12-03 | 10 | 2 | 0.5000 | 0.2399 | 2.3989 |
2022-12-02 | 10 | 3 | 0.3333 | 0.1599 | 1.5993 |
2022-12-01 | 10 | 4 | 0.2500 | 0.1199 | 1.1995 |
2020-01-01 | 1 | 1069 | 0.0009 | 0.0004 | 0.0004 |
- Finally, we compute the value of my custom moving average
$$ MA_5 = \sum ValueAdj $$ $$ MA_5 = 9.9960 $$ $$ \alpha = 1 $$ In which, we can see that the old observation barely affects the outcome. If I wanted the time decay to be weaker, however, all I have to do is adjust $\alpha$.
$$ MA_5 = 8.7284 $$ $$ \alpha = 2 $$ $$ MA_5 = 8.2795 $$ $$ \alpha = 10 $$
This seems to me like a novel solution to my problem. It lets me analyze my data over time if I'm willing to compute this moving average many times while shifting the cutoff date by, say, a day each time. Not to mention, this moving average doesn't freak out if you have many points that occurred at the exact same time, or a singular point that occurred a very long time ago.
My Questions
- Does this type of moving average exist already?
- Does this type of moving average remind you of another one that is more efficient?
- Are there any cases where this moving average behaves unexpectedly?
As I got to coding this solution in Python, I figured it would be a good idea to ask about it here first. I welcome any criticisms or anyone who's got a more elegant solution. Your thoughts are much appreciated, thanks.