# The mathematics behind Adam Optimizer

The PDF link is below: https://arxiv.org/pdf/1412.6980.pdf

The section 2.1 gives the explanation and the intuition in ADAM, but the statements does not make sense for me.

At first, it claims that $| \Delta_t | \le \alpha \cdot (1-\beta_1) / \sqrt{1-\beta_2}$, if $(1-\beta_1)>\sqrt{1-\beta_2}$

In the suggested value of $\beta_1=0.9$, $\beta_2=0.999$, it is in the case.

Latter it claims that $| \Delta_t | \le \alpha$, in common scenarios, but I think that if $\beta_1$, $\beta_2$ is fixed, if would never be in this so called "common scenarios".

Did I miss something? I am really confused.

Reading the paper as it is, I'm fairly sure their upper bounds here are incorrect. For instance, consider the counter example $g_1 = 2$, $g_2 = 1$, $\beta_1 = \beta_2 = 0.5$, $\alpha = 1$, $\epsilon \rightarrow 0$.

Then $m_1 = 1$, $v_1 = 1$, $m_2 = 0.5 \cdot 1 + 0.5 \cdot 1 = 1$ and $v_2 = 0.5 \cdot 1 + 0.5 \cdot 1 = 1$

Therefore, $\widehat{m}_2 = 1/(1 - 1/4) = 4/3$ and $\widehat{v}_2 = 1/(1 - 1/4) = 4/3$ and $|\Delta_2| = 1\cdot (4/3)/\sqrt{4/3} = \sqrt{4/3}$, which is more than both suggested upper bounds.

It's possible that this is a typo in favor of $\alpha \cdot \sqrt{1 - \beta_2}/(1 - \beta_1)$, which seems to have a lot more going for it.

• If $g_1=2$, then $g_1^2=4$ and $v_1=0+0.5\cdot 4 = 2$ and $v_2=0.5 \cdot 2 + 0.5 \cdot 1^2=1.5$
– Alex
Commented Aug 3, 2020 at 16:54
• My previous comment was wrong - with the correct values, the bounds are correct. Commented Mar 22, 2023 at 17:43

My understanding is that those two expressions are two possible upper bounds (applicable depending the situation).

According to the paper (https://arxiv.org/pdf/1412.6980.pdf)

The first case only happens in the most severe case of sparsity: when a gradient has been zero at all timesteps except at the current timestep.

In that situation the upper bound will be:

$| \Delta_t | \le \alpha \cdot (1-\beta_1) / \sqrt{1-\beta_2}$

Otherwise the upper bound will be:

$|\Delta_t | \le \alpha$

• There's a question arose, why does it has these two upper bounds? Commented Jun 11, 2017 at 3:45
• @user3148602 I believe these bounds are wrong, as I tried very hard to derive it but did not succeed, how on earth the authors are allowed to skip such non-trivial bounds??? Commented Oct 9, 2021 at 22:21
• @MathandYuGiOhlover did you get any response? Do you have an update? Empirically these bounds do seem to be right. Commented Mar 22, 2023 at 17:42