Show that $|N_G:G_\alpha|=|\{\beta \in \Omega:g(\beta)=\beta,\forall g \in G_\alpha\}|$ Let $G<S(\Omega)$ be a transitive group and $\alpha \in \Omega$. Show that $$|N_G(G_\alpha):G_\alpha|=|\{\beta \in \Omega:g(\beta)=\beta, \forall g \in G_\alpha\}|.$$
I tried to use Burnside's theorem but all I got was,
that $|N_G(G_\alpha)|=k \cdot |G_\alpha|$ and $k\mid |\Omega|.$
I also noticed that if $\beta \in \Omega:g(\beta)=\beta,\forall g \in G_\alpha\ $ then $h \in N_G(G_\alpha)$ only if $h(\beta)=\beta$, but this gives me nothing
 A: For a subgroup $H$ of $G \le{\rm Sym}(\Omega)$, we define the fixed point set of $H$ to be ${\rm Fix}(H) = \{\beta \in \Omega : h(\beta) = \beta,\,\forall h \in H \}$.
There is a standard result that $N_G(H)$ stabilizes the set ${\rm Fix}(H)$. That is, $g(\beta) \in {\rm Fix}(H)$ for all $\beta \in {\rm Fix}(H)$ and $g \in N_G(H)$. (If you haven't come across that result then try and prove it.)
Now the set on the right hand side of the equality you are trying to prove is just ${\rm Fix}(G_\alpha)$. So is it stabilized by $N_G(G_\alpha)$.
Now the result will follow from the Orbit-Stabilizer Theorem if we can prove that $N_G(G_\alpha)$ acts transitively on ${\rm Fix}(G_\alpha)$.
To do that, let $\beta \in {\rm Fix}(G_\alpha)$. Since $G$ is transitive on $\Omega$, there exists $g \in G$ with $g(\alpha) = \beta$. So $gG_\alpha g^{-1} = G_\beta$. But $\beta \in {\rm Fix}(G_\alpha)$ means that $G_\alpha \le G_\beta$. So, since $\Omega$ is finite, we must have $G_\alpha = G_\beta$, and hence $g \in N_G(G_\alpha)$, proving the result.
(This argument would fail if $\Omega$ was allowed to be infinite, because in that case $G_\alpha$ could be properly contained in $G_\beta$.)
Added later: For the record, this is not true in general without some finiteness hypothesis. Let $H$ be the group consisting of the finitary permutations of a countably infinite set $\Delta$, let $\alpha \in \Delta$. Then $H \cong H_\alpha$, and so we can use an isomorphism $i:H \to H_\alpha$ to define an HNN extension $G=\langle H,t \mid t^{-1}ht = i(t)\,\forall h \in H \rangle$. Then $N_G(H) = H$ but, in the permutation representation of $G$ on the cosets of $H$, the set ${\rm Fix}(H)$ is infinite.
A: More generally, if $G_\alpha\leq X\leq G$, then we can define $$f:X/G_\alpha\to X(\alpha)$$ (from the set of left cosets of $G_\alpha$ in $H$ to the orbit of $\alpha$ under $X$) via
$$f(xG_\alpha)=x(\alpha).$$
One can check that this is a well-defined bijection, so $|X/G_\alpha|=|X(\alpha)|$.
This works even in the infinite case. It remains to check that, if we take $X=N_G(G_\alpha)$, then $X(\alpha)=\mathrm{Fix}(G_\alpha)$, which boils down to showing that $N_G(G_\alpha)$ acts transitively on $\mathrm{Fix}(G_\alpha)$, as in Derek's answer. Here I think you need some kind of finiteness hypothesis. ($\Omega$ finite will do, but there are weaker options as well.)
