The key property you need for the definition to be well-defined is that $f((gh) \cdot x) = f(g \cdot (h \cdot x))$, since $(gh) \cdot x = g \cdot (h \cdot x)$ by the definition of a $G$-action on $X$. Using the definition given, we see that
$$f(g \cdot (h \cdot x)) = f(h \cdot x) \cdot g^{-1} = (f(x) \cdot h^{-1}) \cdot g^{-1} \\ = f(x) \cdot (h^{-1} g^{-1}) = f(x) \cdot (gh)^{-1} = f((gh) \cdot x).$$
Note that we used the definition of a $G$-action on $Y$ in the derivation.
With a similar calculation, you'll see that trying to define equivariance by $f(g \cdot x) = f(x) \cdot g$ fails to be well-defined, i.e. $f((gh) \cdot x) \neq f(g \cdot (h \cdot x))$.