We can assume $\alpha < 1/4$ because otherwise we can apply the trivial bound $|S_f(N)| \leq N$.
Let's partition the sum as follows:
\begin{align*}
\sum_{ 1\leq \ell < N} \min(2N,\| 2\alpha \ell \| ^{-1}) &= \sum_{\substack{1\leq \ell < N \\ 2\alpha\ell < 1/2}} \min(2N,\| 2\alpha \ell \| ^{-1})\\
&+ \sum_{1 \leq m < 2\alpha(N-1)} \sum_{\substack{1\leq \ell < N \\ m-1/2 \leq 2\alpha\ell < m+1/2}} \min(2N,\| 2\alpha \ell \| ^{-1})
\end{align*}
We have the inequalities
\begin{align*}
\sum_{\substack{1\leq \ell < N \\ 2\alpha\ell < 1/2}} \min(2N,\| 2\alpha \ell \| ^{-1}) &\leq \frac{1}{2\alpha} + \frac{1}{2\alpha} \int_{1/2\alpha}^{1/2} \frac{1}{x} \, dx\\
&= \frac{1}{2\alpha} + \frac{1}{2\alpha} \log \frac{1}{4\alpha}
\end{align*}
and
\begin{align*}
\sum_{\substack{1\leq \ell < N \\ m-1/2 \leq 2\alpha\ell < m+1/2}} \min(2N,\| 2\alpha \ell \| ^{-1})) &\leq 2N + 2 \left(\frac{1}{2\alpha} + \frac{1}{2\alpha} \int_{1/2\alpha}^{1/2} \frac{1}{x} \, dx \right)\\
&= 2N + \frac{1}{\alpha} + \frac{1}{\alpha} \log \frac{1}{4\alpha}.
\end{align*}
Therefore we have
\begin{align*}
|S_f(N)|^2 &\leq N + \left(\frac{1}{2\alpha} + \frac{1}{2\alpha} \log \frac{1}{4\alpha} \right) + 2\alpha(N-1)\left(2N + \frac{1}{\alpha} + \frac{1}{\alpha} \log \frac{1}{4\alpha} \right)\\
&\leq 4 \alpha N^2 + 4N \log \frac{1}{\alpha} + \frac{1}{\alpha} \left( \log \frac{1}{\alpha} \right)^2\\
&= \left(2 \sqrt{\alpha} N + \frac{1}{\sqrt{\alpha}} \log \frac{1}{\alpha} \right)^2
\end{align*}
where the second inequality can be shown after a bit of work (using $\alpha < 1/4$).
