What you are asking for is the theory of hyperbolic 2-dimensional orbifolds. It's a very big theory. If you do not add additional hypotheses, it becomes somewhat unreasonable to expect a good description of the theory. Even if you add enough hypotheses to tame the question, it still requires a lot of mathematics to even describe the classification.
Let me tame the question a bit by adding some hypotheses and putting some restrictions on the classification, just in order to give a taste of the big picture.
First, I'll add the hypothesis that the group $\Gamma$ is finitely generated. Next, I'll add a stronger hypothesis of co-compactness, meaning $\Gamma$ has a compact fundamental domain. Co-finite area is also a good hypothesis, as said in my comment, but is trickier to describe correctly, although not really any deeper mathematically. So I am focussing on discrete, co-compact groups $\Gamma < \text{Isom}(\mathbb{H}^2)$ (which, as a consequence, side-steps any questions about analogues of Frieze groups).
The basic level is classification up to isomorphism: the group $\Gamma$ is determined by its quotient orbifold $M_\Gamma = \text{H}^2 / \Gamma$, in the sense that $\Gamma,\Gamma' < \text{Isom}(\mathbb{H}^2)$ are isomorphic if and only if their quotient hyperbolic orbifolds $M_\Gamma,M_{\Gamma'}$ are "orbifold equivalent", meaning that there is a homeomorphism $M_\Gamma \to M_{\Gamma'}$ which preserves the orbifold structure.
On this basic level, the classification in the hyperbolic case is identical (in theory) to the Euclidean and spherical cases: the groups $\Gamma$ with above hypotheses care classified by the closed hyperbolic orbifolds; the 17 wallpaper groups are classified up to isomorphism by the 17 closed Euclidean orbifolds up to orbifold equivalence; and there is a similar spherical classification.
There are infinitely many distinct closed hyperbolic 2-orbifolds $\mathcal{O}$ up to equivalence, and hence infinitely many groups $\Gamma$ up to isomorphism. It is possible to completely list all closed hyperbolic 2-orbifolds $\mathcal{O}$ by keeping track of certain invariants:
- the topological type of the surface $S$ underlying $\mathcal{O}$ (topological Euler characteristic $\chi(S)$, orientability of $S$, number of boundary components of $S$);
- the numbers of cone points of all angles in the interior of $S$;
- the cycles of dihedral points going around each boundary component of $S$ (keeping track of angles of the dihedral points in the cycle, and taking appropriate care of orientation of boundary components depending on whether $S$ itself is orientable).
Of course one must also be sure to discard from this list the spherical, Euclidean, and bad 2-orbifolds, but that can be done using the sign of the orbifold Euler characteristic $\chi(\mathcal{O})$, which can be computed by a formula using the invariants listed above, with the following outcome:
- spherical or bad is equivalent to $\chi(\mathcal{O})>0$;
- Euclidean is equivalent to $\chi(\mathcal{O})=0$;
- hyperbolic is equivalent to $\chi(\mathcal{O})<0$.
The link provided in the first line of the answer gives a complete list of the Euclidean, spherical, and bad cases; everything else is hyperbolic.
You have asked for references. One reference is anything about Conway's orbifold notation. You might also look up any references on Fuchsian groups, although this will cover only the cases where $\Gamma$ preserves orientation (in particular, no reflections). Also, the literature on Fuchsian groups tends to focus even more strictly on the case where the groups have no finite order elements (no rotations), but still one can find some references without that restriction.
