The definition of $S_K$ is complicated, it is hard to figure out a full asymptotic expansion for it.
Let us look at a simpler one which we do know.
For any integer $p > 0$, consider following sum:
$$F(p) = \sum_{k=1}^p \{ \sqrt{k} \}
= \sum_{k=1}^p \sqrt{k} - \sum_{k=1}^p \lfloor \sqrt{k}\rfloor\tag{*1}
$$
Let $x = \sqrt{p}$, $t = \{ x \}$ and $m = x - t = \lfloor x \rfloor$.
For the first term on RHS of $(*1)$, it has following asymptotic expansion:
$$\sum_{k=1}^p \sqrt{k} \asymp
\frac23 x^3 + \frac12 x + \zeta\left(-\frac12\right) +
\frac{1}{24} x^{-1} -\frac{1}{1920} x^{-5} + \frac {1}{9216} x^{-9} +\cdots\tag{*2}
$$
See here for one way to derive this expansion.
For the second term, we can rewrite it as
$$\begin{align}
\sum_{k=1}^p \lfloor \sqrt{k}\rfloor
&= \sum_{k=m^2}^p m + \sum_{\ell=1}^{m-1}\sum_{k=\ell^2}^{(\ell+1)^2-1}\ell\\
&= (p - m^2 + 1)m + \sum_{\ell=1}^{m-1} (2\ell+1)\ell
= (p - m^2 + 1)m + \frac16 (m-1)m(4m+1)\\
&= (x-t)\left(x^2 - (x-t)^2 + 1 + \frac16(x-t-1)(4(x-t)+1)\right)
\end{align}\tag{*3}
$$
Substitute $(*2)$ and $(*3)$ into $(*1)$ and simplify, we obtain
$$F(p) = \frac12 p + A(t)\sqrt{p} + B(t) + \frac{1}{24} p^{-1/2} + \cdots\quad\text{where}\quad
\begin{cases}
A(t) = -(\frac13 + t - t^2)\\
B(t) = \zeta\left(-\frac12\right) + \frac56 t + \frac12 t - \frac13 t^3
\end{cases}
$$
Since $\displaystyle\;\int_0^1 A(t) dt = -\frac12 \ne 0$, even after
we subtract away the piece $-\frac12 p$, the sum
$$\sum_{k=1}^p \left( \{ \sqrt{k} \} - \frac12 \right) = F(p) - \frac12 p$$
is still systematically biased by a term of the form $-\frac12 \sqrt{p}$. This is the source of the $O(1)$ term you notice.
To see this, let us rewrite the sum at hand in terms of $F(p)$.
Let $\theta_K = \#\{ n \in \mathbb{Z}_{+} : \sqrt{K} \le n \le 2\sqrt{K} \}$,
the sum at hand can be rewritten as
$$\sum_{K=1}^N \left( S_K - \frac{1}{2}\sqrt{K} \right)
= \underbrace{\sum_{K=1}^N \sum_{\sqrt{K} \le n \le 2\sqrt{K}}\left(\left\{ \sqrt{n^2 - K} \right\} - \frac12\right)}_{\mathcal{I}}
- \underbrace{\frac12 \sum_{K=1}^N \left(\sqrt{K} - \theta_K\right)}_{\mathcal{J}}
$$
For the first term $\mathcal{I}$, we can express it using at most two $F(\cdot)$.
$$\begin{align}
\mathcal{I} &= \sum_{n=1}^{2\sqrt{N}}\sum_{\frac14 n^2 \le K \le \min( N, n^2 )}
\left(\left\{ \sqrt{n^2 - K} \right\} - \frac12\right)\\
&= \sum_{n=1}^{2\sqrt{N}}\sum_{ \max(0,n^2 - N)\le \ell \le \frac34 n^2}
\left(\left\{ \sqrt{\ell} \right\} - \frac12\right)\\
&= -\frac12\lfloor\sqrt{N}\rfloor + \sum_{n=1}^{2\sqrt{N}}\sum_{ \max(1,n^2 - N)\le \ell \le \frac34 n^2}
\left(\left\{ \sqrt{\ell} \right\} - \frac12\right) \\
&= -\frac12\lfloor\sqrt{N}\rfloor + \sum_{n=1}^{2\sqrt{N}}\left[ F(p) - \frac12 p \right]_{p = \max(0,n^2-N-1)}^{\lfloor \frac34 n^2\rfloor}
\end{align}
$$
For the second term $\mathcal{J}$, one can show
$$\sum_{K=1}^p \theta_K = \sum_{K=1}^p \left(\lfloor\sqrt{K}\rfloor + \frac12\right) + \frac12 | p - m(m+1) |$$
This implies $\mathcal{J}$ falls of like $\sqrt{N}$ and one can ignore them
in the final limit
$$\mathcal{J} = \frac12\sum_{K=1}^N \left(\sqrt{K} - \theta_K \right)
= \frac12( A(t) - \frac12|1-2t|)\sqrt{N} + O(1)\quad\text{ where }\quad t = \{\sqrt{N}\}$$
To proceed further, we will wave our hands. We will assume
- only the leading term $A(t)\sqrt{p}$ in $F(p) - \frac12 p$ matters.
- we can average out $A(t)$ over $t$, i.e. replace $A(t)\sqrt{p}$ by $-\frac12\sqrt{p}$
- we can approximate the sum over $n$ by an integral over $n$
Changing variable to $n = \sqrt{N}s$, we find
$$\begin{align}\frac1N\sum_{K=1}^N \left( S_K- \frac{1}{2} \sqrt{K} \right)
= \frac{\mathcal{I}}{N} + O (N^{-1/2})
&\approx
-\frac{1}{2N} \left[
\int_0^{2\sqrt{N}} \frac{\sqrt{3}}{2} n dn
- \int_{\sqrt{N}}^{2\sqrt{N}}\sqrt{n^2-N} dn\right]\\
&= -\frac{1}{2}
\left[ \frac{\sqrt{3}}{2} \int_0^{2} s ds
- \int_1^2 \sqrt{s^2 - 1}ds \right]\\
&= -\frac14\log(2+\sqrt{3}) \approx -0.3292394742312041
\end{align}
$$
A number compatible with what OP described in question.
