We can find a range based on condition.
From the first equation, we have that:
$$x+y + [x]+[y] = 5.1\\
2(x+y) - (f_x+f_y) = 5.1$$
Based on the range of $f$ which lies in $[0,1)$, we can say that:
$$x+y \in \left[\frac{5.1}{2}, \frac{7.1}{2}\right)$$
Similarly from second equation, one can deduce:
$$x^2 + y^2 \in \left[\frac{50.93}{2}, \frac{52.93}{2}\right)$$
Basically the first condition gives two lines and second condition gives two circles. Solution lies in area trapped between these curves!
Finding the solution
What I am saying is that you can find solution analytically. Consider the first area. You can find the coordinates of intersection of circle and straight line analytically. I am graphing it to show that the range is quite small:

We are lucky that here, $y$ only lies in $(4,5)$ so that $[y] = 4$. But $x$ lies in $(-3,-1)$, so you must consider two intervals for fixing $[x]$.
$x\in (-3, -2)$. So here, $[x] = -3$. This gives us $x+y = 4.1$, which is not possible.
$x \in [-2,-1)$. So here, $[x] = -2$. This gives us $x+y = 3.1$, which is possible.
We have narrowed down to $[x] = -2$ and $[y] = 4$. Now we have more cases for $[x^2]$ and $[y^2]$! Using our conditions we get $1\le [x^2]\le 4$ and that $16 \le [y^2] \lt 25$
Using the second equation, you must have $[x^2] + [y^2] \in (24.465,25.465)$. Thus $[x^2] + [y^2] = 25$. From this we get only three possible pairs of $([x^2],[y^2])$, which are $(1,24), (2,23), (3,22)$
Thus we only deal with these three cases and you will be able to solve it analytically!
Also noting that the system is symmetric in $x,y$ so that we have an even number of roots.