$\text{SL}_2(\mathbb Z)$ acts on upper plane $\mathbb H$. What kind of points have non-trivial stabilizer? And how many orbits are there? $\text{SL}_2(\mathbb Z)$ acts on upper plane $\mathbb H= \{z \in \mathbb{C} | \Im(z) > 0 \}$ via Mobius transformation.  $$
    \text{ For } \gamma =\begin{bmatrix} 
    a &b \\c&d \end{bmatrix} \in\text{SL}_2(\mathbb Z), \ \gamma z =\begin{bmatrix} 
    a &b \\c&d \end{bmatrix}\cdot z = \frac{az +b}{cz+d}   $$ 
Stabilizer of $z$ means set $\{\gamma \in \text{SL}_2(\Bbb Z), \gamma z=z\}$.
I want to know what kind of points have non-trivial stabilizer and the number of orbits.
My effort:
For $z \in \Bbb H$, suppose $z = x + i y$. 
$\text{For } \gamma =\begin{bmatrix} a &b \\c&d \end{bmatrix}, \gamma z =\frac{az+b}{cz+d}=z\iff az+b=cz^2+dz$
$$\iff ax+ayi+b = cx^2-cy^2+2cxyi + dx +dyi$$
$$\iff ay=2cxy+dy\ \&\  ax+b=cx^2-cy^2+dx$$
$$\iff a=2cx+d \ \&\  ax+b=cx^2-cy^2+dx$$
$$\implies b=-c(x^2+y^2),\ \gamma =\begin{bmatrix} 2cx+d &-c(x^2+y^2) \\c&d \end{bmatrix}.$$
$$\gamma \in \text{SL}_2(\Bbb Z),\ (2cx+d)\times d-(-c(x^2+y^2))\times c=1 $$
$$\implies (cx+d)^2+(cy)^2=1.$$
Then how to proceed? Thanks in advance.
I haven't learnt modular form yet, and I don't know if these help:
Good description of orbits of upper half plane under $SL_2 (Z)$
Orbit of complex unit $i$ under moebius tranformation in $SL_2(\mathbb{Z})$
Edit:
GTM$105$, Serge Lang, SL$_2(\mathbb R)$ might help.
Comment:
It's an exercise of section about group action on set, and before this section the book just introduces definition and basic property of group, so this problem is a bit more difficult than I thought.
 A: The group $\mathrm{PSL}_2(\mathbf{Z})$ is a free product $C_2\ast C_3$, and hence its nontrivial torsion elements have order 2 or 3, and has 1 conjugacy class of cyclic subgroup of order 2 (a representative is $z\mapsto -1/z$ with fixed point $i$), 1 conjugacy class of cyclic subgroup of order 3 (a representative being generated by $z\mapsto -1/(1+z)$, fixing $j$, the unique root of $1+z+z^2$ in the upper half-plane). The elements of infinite order have no fixed point on $\mathbf{H}$. 
So the points with nontrivial stabilizer are those in the orbit of $i$ and the orbit of $j$. They are not in the same orbit since otherwise the stabilizer should contain an element of order 6.
A: At some point you have $b=-c(x^2+y^2)$ and $a=2cx+d$.  If $c=0$, then $b=0$, in which case the element $\begin{pmatrix} a&b\\c&d\end{pmatrix}$ of the projective special linear group  $\operatorname{PSL}_2(\Bbb Z)$ of this form is the identity element (recalling that $\operatorname{PSL}_2(\Bbb Z)=\operatorname{SL}_2(\Bbb{Z})/\{\pm I\}$), so we are not interested in this case.  So, $c$ is assumed to be non-zero, and can without loss of generality assumed to be positive.
That is, $x=\frac{a-d}{2c}$, so that $$-\frac{b}{c}=x^2+y^2=\frac{(a-d)^2}{4c^2}+y^2.$$
Since $y>0$, we have
$$y=\frac{\sqrt{4-(a+d)^2}}{2c},$$ which means $d=-a$ or $d=-a\pm1$.  
In the case $d=-a$, we must have $x=\frac{a}{c}$ and $y=\frac{1}{c}$.
For a given point $z=x+yi$ of this form that is fixed by a non-trivial $\gamma\in\operatorname{PSL}_2(\Bbb{Z})$, we can assume that $$\gamma=\begin{pmatrix}a&b\\c&-a\end{pmatrix}\wedge bc=-(a^2+1).\tag{1}$$  That is, $\gamma^2$ is the identity of $\operatorname{PSL}_2(\Bbb{Z})$.  The points in $\Bbb H$ with a stabilizer of order $2$ are of the form $z=x+yi$, where
$$x=\frac{a}{c}\wedge y=\frac{1}{2c}$$
for some integer $a$ and for some integer $c>0$, such that $a^2+1$ is divisible by $c$ (so that there exists $b\in \Bbb Z$ such that $bc=-(a^2+1)$).  The number of orbits for a given $a$ is $\sigma_0(a^2+1)$, where $\sigma_0$ is the divisor counting function.
For the case $d=-a+1$, we see that $x=\frac{2a-1}{2c}$ and $y=\frac{\sqrt{3}}{2c}$.  Then, $z=x+yi$ is fixed by
$$\gamma=\begin{pmatrix}a&b\\c&-a+1\end{pmatrix}\wedge bc=-(a^2-a+1).\tag{2}$$
Note that $\gamma^3$ is the identity of $\operatorname{PSL}_2(\Bbb{Z})$.  Thus, for a given $a$, there are corresponding $\sigma_0(a^2-a+1)$ points $z\in \Bbb H$.
For the case $d=-a-1$, we see that $x=\frac{2a+1}{2c}$ and $y=\frac{\sqrt{3}}{2c}$.  Then, $z=x+yi$ is fixed by
$$\gamma=\begin{pmatrix}a&b\\c&-a-1\end{pmatrix}\wedge bc=-(a^2+a+1).\tag{3}$$ 
Note that $\gamma^3$ is the identity of $\operatorname{PSL}_2(\Bbb{Z})$.  Thus, for a given $a$, there are corresponding $\sigma_0(a^2+a+1)$ points $z\in \Bbb H$.  (It can be seen that this case is identical to the previous case via the transformation $a\mapsto a-1$.)
For example, $x=\frac{3}{5}$ and $y=\frac{1}{5}$ fit the bill (with $a=3$, $b=-2$, $c=5$, and $d=-a=-3$) for a point with stabilizer of order $2$, with $$\gamma=\begin{pmatrix}3&-2\\5&-3\end{pmatrix}.$$
For the same $a=3$, there are three more points with stablizers of order $2$, i.e., with $c=1$, $c=2$, and $c=10$.  That is, for $a=3$, we have in total four points with non-trivial stabilizers $\gamma$ of order $2$: $z=3+i$ with $\gamma=\begin{pmatrix}3&-10\\1&-3\end{pmatrix}$, $z=\frac{3}{2}+\frac{i}{2}$ with $\gamma=\begin{pmatrix}3&-5\\2&-3\end{pmatrix}$, $z=\frac{3}{5}+\frac{i}{5}$ with $\gamma=\begin{pmatrix}3&-2\\5&-3\end{pmatrix}$, and $z=\frac{3}{10}+\frac{i}{10}$ with $\gamma=\begin{pmatrix}3&-1\\10&-3\end{pmatrix}$. 
For the same $a=3$, there are also four points with stabilizers of order $3$.  That is, for $a=3$, we have four points with non-trivial stabilizers $\gamma$ of order $3$: $z=\frac52+\frac{\sqrt{3}i}{2}$ with $\gamma=\begin{pmatrix}3&-7\\1&-2\end{pmatrix}$, $z=\frac{5}{14}+\frac{\sqrt{13}i}{14}$ with $\gamma=\begin{pmatrix}3&-1\\7&-2\end{pmatrix}$, $z=\frac{7}{2}+\frac{\sqrt{3}i}{2}$ with $\gamma=\begin{pmatrix}3&-13\\1&-4\end{pmatrix}$, and $z=\frac{7}{26}+\frac{\sqrt{3}i}{26}$ with $\gamma=\begin{pmatrix}3&-1\\13&-4\end{pmatrix}$. 
In conclusion, there are three kinds of points $z\in\Bbb H$---those with trivial stabilizers, those with stabilizers of order $2$, and those with stabilizers of order $3$.  The stablizers in non-trivial cases are generated by $\gamma$ given in (1) and (2).   
A: I just realized that the points with stabilizers of order $2$ are in the same orbit, and the points with stabilizers of order $3$ are in the same orbit.  There are therefore exactly two orbits with non-trivial stabilizers.  The previous answer is too long and the browser is becoming slow, so I have to add another answer.
In the case of stabilizers of order $2$, recall that $c\mid a^2+1$.  Using the knowledge about Gaussian integers (in particular, the unique factorization property), $c$ can be written as $$r^2+s^2=(r+si)(r-si)$$ for some $r,s\in\Bbb{Z}$ such that $r-si$ divides $a+i$.  That is, $$\frac{a+i}{r-si}=p+qi$$ for some $p,q\in\Bbb{Z}$.  Then observe that $$\begin{pmatrix}q&p\\s&r\end{pmatrix}\cdot i=\frac{p+qi}{r+si}=\frac{(p+qi)(r-si)}{r^2+s^2}=\frac{a+i}{r^2+s^2}=\frac{a}{c}+\frac{i}{c}.$$
Thus, $\frac{a}{c}+\frac{i}{c}$ is in the orbit of $i$.
In the case of stabilizers of order $3$, we can assume WLOG that we are in the case $d=-a+1$, where $c\mid a^2-a+1$.  Let $\omega$ denote $-\frac12+\frac{\sqrt{3}i}{2}$.  Using the knowledge about Eisenstein integers (in particular, the unique factorization property), $c$ can be written as $$r^2-rs+s^2=(r+s\omega)(r+s\overline{\omega})$$ for some $r,s\in \Bbb Z$ such that $r+s\overline{\omega}$ divides $a+\omega$.  That is, $$\frac{a+\omega}{r+s\overline{\omega}}=p+q\omega$$ for some $p,q\in \Bbb Z$.  Finally, observe that
$$\begin{pmatrix}q&p\\s&r\end{pmatrix}\cdot\omega=\frac{p+q\omega}{r+s\omega}=\frac{(p+q\omega)(r+s\overline{\omega})}{r^2-rs+s^2}=\frac{a+\omega}{c}=\frac{2a-1}{2c}+\frac{\sqrt{3}i}{2c}.$$
Thus, $\frac{2a-1}{2c}+\frac{\sqrt{3}i}{2c}$ is in the orbit of $\omega$.
