An element conjugate to an odd number of elements

I have the following problem I've encountered while studying for quals that I just can't seem to tackle:

(revised, apologies - left out an important detail)

An element g of order greater than 2 such that the conjugacy class of G has an odd number of elements. Prove that g is not conjugate to $g^{-1}$.

-

If $g$ is conjugate to $g^{-1}$, then whenever $g$ is conjugate to $h$ it is also conjugate to $h^{-1}$. If $g \ne g^{-1}$, the elements conjugate to $g$ (including, of course, $g$ itself) split up into pairs $(h, h^{-1})$. However, if $g = g^{-1}$, $g$ could be conjugate to any number of elements (which, of course, are each equal to their inverses).
Another way to look at it is that you are told that $[G:C_{G}(g)]$ is odd, as this is the number of conjugates of $g.$ Hence $C_{G}(g)$ contains a Sylow $2$-subgroup of $G$. Let $H = C^{*}_{G}(g),$ the so-called extended centralizer of $g,$ that is $H = \{ x \in G: x^{-1}gx \in \{g,g^{-1} \} \}.$ (This subgroup has been studied by R. Brauer among others. It is indeed a subgroup of $G$ which contains $C_{G}(g)$ and is contained in $N_{G}(\langle g \rangle)).$ Note that $[H:C_{G}(g)] \leq 2,$ since every $H$-conjugate of $g$ is either $g$ or $g^{-1}$ by definition of $H.$ But $C_{G}(g)$ already contains a Sylow $2$-subgroup of $G,$ so we can't have $[H:C_{G}(g)] = 2.$ Hence we must have $H = C_{G}(g)$ under the hypotheses of the question. Thus the only way $g$ can be conjugate to $g^{-1}$ in $G$ is if $g = g^{-1}$ already, that is, $g$ has order $1$ or $2$.