1
$\begingroup$

According to this and many other places, weight for exponential moving average is just being defined as $\omega_t=(1-\alpha)\alpha^t$, where $t$ is current index and $\alpha$ is a smoothing factor.

How does one derives this formula itself and what does $\alpha$ mean, and where does one can plug size of averaging window?

This is the problem for me as I expected $\omega$ to be a function of window size $N$ and index $t$, but here and everywhere else I got only $t$ and mysterious $\alpha$.

I understand that $0<\alpha<1$ and that it describes the steepness of the exponential slope, but I am confused that I cant find the derivation of this formula. That is why I cant understand it to the end. Could anybody provide step by step derivation of this?

$\endgroup$

2 Answers 2

1
$\begingroup$

With exponential moving average, your averaging window includes all previous values, although most recent values weight more. A finite w can not thus be defined in this case.

On the other hand, you can select $\alpha$ so that the last w samples make up for a given portion of your current estimate.

In your discrete case, an $\alpha$ value such that the last w samples make up for about 62.3% of the current estimate would be:

$$ \alpha = 1 - e^{(-1/w)} $$

https://en.wikipedia.org/wiki/Exponential_smoothing#Time_Constant

$\endgroup$
0
$\begingroup$

The weight can be derived when you view the EMA (Exponential Moving Average) update as a difference equation:

$$ y_t = \alpha y_{t-1} + (1-\alpha) x_t $$

Here, $(1-\alpha)$ is a measure of how much you "trust" the new observation $x_t$: the higher $\alpha$, the more you tend to stick to the accumulated average $y_{t-1}$, and the less the impact $x_t$ has on the new average $y_t$.

In a usual average of two quantities their weights are equal to $1/2$:

$$ \operatorname{avg}(y_{t-1}, x_t) = \frac{y_{t-1} + x_t}2 = \frac12 y_{t-1} + \frac12 x_t $$

With EMA, you use $\alpha \in (0, 1)$ for more precise control of the balance between $y_{t-1}$ and $x_t$.

One can solve this difference equation to obtain the solution which is (almost) the weighted sum as in the question.

Solving the difference equation

The difference equation of the EMA can be solved like this. First, manually calculate $y_t$ for a few values of $t$:

$$ \begin{aligned} y_1 &= \alpha y_0 + (1-\alpha) x_1\\ y_2 &= \alpha y_1 + (1-\alpha) x_2 = \alpha^2 y_0 + \alpha(1-\alpha) x_1 + (1-\alpha) x_2\\ y_3 &= \alpha y_2 + (1-\alpha) x_3 = \alpha^3 y_0 + \alpha^2(1-\alpha) x_1 + \alpha (1-\alpha) x_2 + (1-\alpha)x_3\\ \end{aligned} $$

It looks like the formula for $y_t$ goes something like this:

$$ y_t = \alpha^t y_0 + \sum_{i=1}^t \alpha^{t-i} (1-\alpha) x_i $$

Alternatively, you can let $i$ represent the power of $\alpha$, so the formula becomes:

$$ y_t = \alpha^t y_0 + \sum_{i=0}^{t-1} \alpha^i (1-\alpha) x_{t-i} $$

Finally, use induction to prove that this solution is correct:

  1. Base case: $y_0=\alpha^0 y_0 + 0$ is true.
  2. Assume $y_t$ is true. Show that $y_{t+1}$ satisfies the difference equation:

$$ \begin{split} y_{t+1} &= \alpha y_t + (1-\alpha) x_{t+1}\\ &= \alpha^{t+1}y_0 + \alpha\sum_{i=0}^{t-1} \alpha^i (1-\alpha) x_{t-i} + (1-\alpha) x_{t+1}\\ &= \alpha^{t+1}y_0 + \alpha (1-\alpha) (\alpha^0 x_t + \alpha^1 x_{t-1} + \dots + \alpha^{t-1} x_{1}) + \alpha^0 (1-\alpha) x_{t+1}\\ &= \alpha^{t+1}y_0 + (1-\alpha) (\alpha^1 x_t + \alpha^2 x_{t-1} + \dots + \alpha^{t} x_{1}) + \alpha^0 (1-\alpha) x_{t+1}\\ &= \alpha^{t+1}y_0 + (1-\alpha) (\alpha^0 x_{t+1} + \alpha^1 x_t + \alpha^2 x_{t-1} + \dots + \alpha^{t} x_{1})\\ &= \alpha^{t+1}y_0 + (1-\alpha) \sum_{i=0}^{t} \alpha^i x_{t+1-i}\\ &= \alpha^{t+1}y_0 + \sum_{i=0}^{t} \alpha^i (1-\alpha) x_{t+1-i} \end{split} $$

Indeed, we get the same formula as for $y_t$, but with $t+1$ instead of $t$ everywhere, so the solution is correct.

The weight

Looking at the formula again:

$$ y_t = \alpha^t y_0 + \sum_{i=0}^{t-1} \alpha^i (1-\alpha) x_{t-i} $$

...we see that it's of this form:

$$ \begin{cases} y_t &= \alpha^t y_0 + \sum_{i=0}^{t-1} w_i x_{t-i}\\ w_i &= (1-\alpha) \alpha^i \end{cases} $$

...where $i$ is the current index and $\alpha$ is the smoothing factor, just like in the question.

Practical consideration

In practice we don't know the initial condition $y_0$ because we only observe $y_t$ starting from $t=1$. However, the weight of $y_0$ is $\alpha^t$, which quickly goes to zero when $\alpha \in (0, 1)$, so one can safely ignore the initial condition and use the sum as the average:

$$ y_t \approx \sum_{i=0}^{t-1} \alpha^i (1-\alpha) x_{t-i} $$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .