$f$ is a bijection from $\Bbb R \setminus \{-2\}$ to $\Bbb R \setminus \{1\}$.
Strictly speaking, it cannot have a functional square root : If $f = g \circ g$, then $g(g(-5/3)) = f(-5/3) = -2$. If $g(-5/3) = -2$ then $g(-2)=-2$ and we get that $g\circ g$ is defined at $-2$ when it should not. So $g(-5/3) \neq -2$ and $f$ is defined at $g(-5/3)$ : $f(g(-5/3)) = g(-2)$. Again, if $f(g(-5/3)) = -2$ then $g(-2) = -2$, so $f$ is defined at $f(g(-5/3))$, and so $g$ has to be defined at $g(-2)$. In any case, $g(g(-2))$ exists while it shouldn't.
To make the question interesting we plug the hole and consider $f$ as a bijection from $X = \Bbb R \cup \{\infty \}$ to itself. $f$ has two fixpoints $0 < x_1 < x_2$, any iterate of $f$ is still an homography so still has only those two same fixpoints. This means that you can partition $X' = X \setminus \{x_1,x_2\}$ into sequences of the form $\{f^k(x), k \in \Bbb Z\}$
If you don't need $g$ to be continuous, there are lots of ways to do define a functional square root. Given any point $x \in X'$, you can choose $g(x)$ to be any point $y \in X'$ as long as $y \notin \{f^k(x), k \in \Bbb Z\}$. This choice will determine $g$ at all the $f^k(x)$ and $f^k(y)$ for $k \in \Bbb Z$. Repeat this until you have defined $g$ on all of $X'$. Finally define either $g(x_i)= x_i$ or $g(x_i) = x_{3-i}$.
If you want $g$ to be continuous, there are two cases to consider. $X'$ has two connected components, $(x_1;x_2)$ and $(x_2; \infty) \cup \{\infty\}\cup\{- \infty ; x_1\}$, which should just be called $(x_2 ; x_1)$. Those two components are stable by $f$. It is useful to put an order on those components such that $f$ is increasing. On the first component it is the standard order, but on the other one, we have $y < \infty < z$ if $y > x_2$ and $z < x_1$.
If $g$ switches the connected components $(x_1;x_2)$ and $(x_2 ; x_1)$ of $X'$, $g$ will be determined by its image on an interval of the form $[x ; f(x))$ where $x \in X'$ : you have to send $x$ to some $y$ (on the other component), send $f(x)$ to $f(y)$, and in-between, $g$ can be any strictly increasing (for the orders defined above) continuous function.
If $g$ doesn't switch them, we have to define $g$ separately on each component. To do that, pick an $x \in X'$, choose some $g(x) \in (x ; f(x))$. Then we must have $g(g(x)) = f(x)$, and we can pick any strictly increasing continuous function on $[x ; g(x)]$. This determines $g$ completely on the component of $x$. do the same on the other component and we are done.
In both cases, for $g$ to be continuous, we must have $g(x_i) = x_i$ if $x_i$ is one of the two fixpoints.
Also, those constructions can give you functional square roots that are smooth on $X'$ (I(m not sure what happens at the ficpoints) and are still not one of the two homographies that you have found.