We have a fixed $\varepsilon \gt 0$ and we want to set $\delta \gt 0$ so that for every $\delta_0 \in (-\delta,+\delta)\;$ (i.e. $|\delta_0| \lt \delta$),
$\tag 1 4- \varepsilon \lt 4 + 4 \delta_0 + {\delta_0}^2 \lt 4 + \varepsilon$
Now if $\delta = min(1, \frac{\varepsilon}{5})$,
$\quad 4 \delta_0 + {\delta_0}^2 \le |4 \delta_0| + |\delta_0| = 5 \, | \delta_0| \lt \varepsilon $
so the inequality on the rhs of $\text{(1)}$ will be true for this setting of $\delta$.
Fortunately, this setting also works for the lhs of $\text{(1)}$:
We have
$\quad -4 \delta_0 \le 4 |\delta_0 | \lt 4 \delta \le \frac{4 \varepsilon} {5} \lt \varepsilon$
But then $-\varepsilon \lt 4 \delta_0 \lt 4 \delta_0 + {\delta_0}^2$.
Note: For the lhs we only need to set $\delta = \frac{\varepsilon}{5}$. I guess we were lucky that we started with the rhs!