There is a very elegant way to prove a very general statement, in relation to which your problem above occurs as a particular case. Consider an arbitrary group $\Gamma$ (no finiteness required) with a left action $\cdot\ \colon \Gamma \times A \rightarrow A$ on an arbitrary set $A$.
Let us adopt the following syntactic (notational) conventions:
- given arbitrary subset $\Lambda \subseteq \Gamma$ of operators and arbitrary $X \subseteq A$, we write:
$$\Lambda X \stackrel{\textrm{def}}{=} \cdot \left[\Lambda \times X\right]=\left\{\lambda x\right\}_{\substack{\lambda \in \Lambda\\x \in X}}$$
to denote the external product of the two subsets (external as it stems from an external operation, the action itself). When either of the two subsets in question is a singleton, we also adopt the abbreviations:
$$\lambda X\stackrel{\textrm{def}}{=}\{\lambda\}X$$
for any $\lambda \in \Gamma$ and $X \subseteq A$ respectively
$$\Lambda x \stackrel{\textrm{def}}{=}\Lambda \{x\}$$
for any $\Lambda \subseteq \Gamma$ and $x \in A$.
- given arbitrary subsets $X, Y \subseteq A$ we write:
$$\left(Y \colon X\right)_\Gamma \stackrel{\textrm{def}}{=} \left\{\lambda \in \Gamma \mid \lambda X \subseteq Y\right\}$$
to refer to the quotient between the two subsets (the subscript $\Gamma$ serves to specify the action with respect to which the quotient is considered). As in the previous convention, we also adopt the abbreviations:
$$\left(Y \colon x\right)_{\Gamma} \stackrel{\textrm{def}}{=} \left(Y \colon \{x\}\right)_{\Gamma}$$
for arbitrary $Y \subseteq A$ and $x \in A$ respectively
$$\left(y\ \colon X\right)_{\Gamma} \stackrel{\textrm{def}}{=}\left(\{y\}\colon X\right)_{\Gamma}$$
for arbitrary $X \subseteq A$ and $y \in A$.
- given arbitrary binary relation $R \in \mathscr{Rel}(A)$ on $A$ together with arbitrary subset $M \subseteq A$, we write $R_{|M} \stackrel{\textrm{def}}{=} R \cap \left(M \times M\right)$ for the restriction to $M$ of relation $R$; if $R \in \mathscr{Eq}(A)$ is an equivalence on $A$ then the restriction $R_{|M} \in \mathscr{Eq}(M)$ is correspondingly an equivalence on $M$. Furthermore, in the particular case when $M$ is saturated with respect to equivalence $R$ - which means that for any given $x \in M$ we automatically have $R\langle x \rangle \subseteq M$, in other words together with any element $M$ contains (includes) the entire $R$-class of that element - the following relation holds:
$$M/R_{|M}=\left\{X \in A/R \mid X \subseteq M\right\}=A/R \cap \mathscr{P}(M).$$
This relation means that the $R_{|M}$-classes of $M$ are none other than the $R$-classes (in the ambient set $A$) included in $M$.
- for arbitrary set $M$ we write $\Delta_M$ for its diagonal and given an arbitrary map $f \colon A \to B$ we denote its canonical equivalence by $\mathrm{Eq}(f)\stackrel{\textrm{def}}{=}\left(f \times f\right)^{-1}\left[\Delta_B\right]$; this equivalence relation can also be described by the (propositional) equivalence:
$$\left(\forall x, y\right)\left(x\mathrm{Eq}(f)y \Leftrightarrow x, y \in A \wedge f(x)=f(y)\right).$$
- given arbitrary group $F$ and subgroup $H \leqslant F$, we shall write $\mathrm{Cg}^F_{\mathrm{s}}(H)$ for the left congruence on $F$ modulo $H$; this is the equivalence relation described by the (propositional) equivalence:
$$x\mathrm{Cg}^F_{\mathrm{s}}(H)y \Leftrightarrow x, y \in G \wedge x^{-1}y \in H.$$
We also write $\left(F/H\right)_{\mathrm{s}}\stackrel{\textrm{def}}{=}F/\mathrm{Cg}^F_{\mathrm{s}}(H)$ to denote the quotient set consisting of all left congruence classes with respect to $H$ (in case anyone might be wondering about the subscript $\mathrm{s}$, it is the initial of the word left in both Latin as well as in my mother tongue).
Returning to the general setting of a given left action of $\Gamma$ on $A$, let us fix $x \in A$ and $X \subseteq A$. We shall establish the following:
Proposition. The cardinal relation $\left\lvert \left(X \colon x\right)_{\Gamma}\right\rvert=\left\lvert \Gamma x \cap X\right\rvert \left\lvert \mathrm{Stab}_{\Gamma}(x)\right\rvert$ holds, where the notation $\mathrm{Stab}$ is the traditional one for stabilisers of elements.
Proof : for simplicity let us introduce the notations $\Lambda \stackrel{\textrm{def}}{=} \left(X \colon x\right)_{\Gamma}$, $H \stackrel{\textrm{def}}{=}\mathrm{Stab}_{\Gamma}(x)$ and $R \stackrel{\textrm{def}}{=} \mathrm{Cg}_{\mathrm{s}}^{\Gamma}(H)$. It is clear that for any $\lambda \in \Lambda$ it is the case that $\lambda x \in \Gamma x \cap X$, which enables us to consider the following map:
$$\begin{align}
\varphi \colon \Lambda &\to \Gamma x \cap X \\
\varphi(\lambda)&=\lambda x.
\end{align}$$
The following are immediate observations:
- $\varphi$ is surjective
- the canonical equivalence of $\varphi$ is the restriction to $\Lambda$ of the left congruence modulo $H$, formally speaking $\mathrm{Eq}\left(\varphi\right)=R_{|\Lambda}$
- $\Lambda$ is saturated with respect to $R$, which means explicitly that given any element $\lambda \in \Lambda$ we have $\lambda H \subseteq \Lambda$, i.e the entire left $H$-class of $\lambda$ (with respect to $R$) is included in $\Lambda$.
By virtue of observation 1) together with the fundamental theorem for quotient maps in set theory, we infer that:
$$\left\lvert \Gamma x \cap X \right\rvert=\left\lvert \Lambda/\mathrm{Eq}\left(\varphi\right) \right\rvert. \tag{i}$$
By virtue of observation 2) and the above we gather that:
$$\left\lvert \Gamma x \cap X \right\rvert=\left\lvert \Lambda/R_{|\Lambda} \right\rvert. \tag{ii}$$
By virtue of observation 3) together with the remark concerning saturated subsets made in one of the above paragraphs, it follows that $\Lambda/R_{|\Lambda} \subseteq \left(\Gamma/H\right)_{\mathrm{s}}$. This entails the fact that:
$$\left(\forall \Theta\right)\left(\Theta \in \Lambda/R_{|\Lambda} \Rightarrow \left\lvert \Theta \right\rvert=\left\lvert H\right\rvert\right), \tag{iii}$$
in other words all the equivalence classes in the quotient set $\Lambda/R_{|\Lambda}$ have cardinality equal to that of $H$, since they are all left $H$-congruence classes of $\Gamma$. Taking into account relations $\textrm{(i)-(iii)}$ above we can produce the following relation:
$$\left\lvert \Lambda \right\rvert=\left\lvert \bigcup \Lambda/R_{|\Lambda} \right\rvert=\sum_{\Theta \in \Lambda/R_{|\Lambda}}\left\lvert \Theta \right\rvert=\sum_{\Theta \in \Lambda/R_{|\Lambda}} \left\lvert H \right\rvert=\left\lvert \Lambda/R_{|\Lambda} \right\rvert \left\lvert H \right\rvert=\left\lvert \Gamma x \cap X \right\rvert \left\lvert H \right\rvert,$$
none other than the relation we had set out to prove. $\Box$
This general proposition has an obvious analogue referring to the dual case of right actions. Your particular formulation of the problem is the special case when $\Gamma=A$ and $\Gamma$ acts on itself by left (or right) conjugation. The subgroup $H$ is in the role of the generic subset $X \subseteq \Gamma$ in my formulation above (at this point it is worthy of remark that the particular subset $H$ - in your notation - being a subgroup plays absolutely no part in the proof) and the stabiliser of any $x \in \Gamma$ with respect to conjugation (whether left or right, regardless) is precisely the centraliser $\mathrm{C}_{\Gamma}(x)$.
To make one more technical remark, the subset featured in the left-hand term of your formulation of the cardinal relation above is - as you define it - the quotient between $X$ and $x$ with respect to right conjugation, so an application of the right-action version of the general proposition is most straight-forward.
