In the paper ADAM they explain how the optimizer work in algorithm 1. In the last step of the while loop they update the parameters with
$$ \theta_{t+1} = \theta_{t} + \alpha \frac{\hat{m}_t}{(\sqrt{\hat{v}}_t - \epsilon)}$$
where $\alpha \in \mathbb{R}$ and I suppose $\theta, \hat{m}_t, \hat{v}_t,\epsilon \in \mathbb{R}^n$ for some $n$. They do not specify the calculation procedure for $\frac{\hat{m}_t}{(\sqrt{\hat{v}}_t - \epsilon)}$. I suppose everything is done element wise but I'm not sure since they specify how they calculate the square gradient by the element wise operation but not the last step.
