# False proof that $\langle\chi,1_G\rangle$ need not be an integer.

I'd like to know where the following calculation has gone wrong. I'm sure it is a silly error.

Let $$G$$ be a finite group acting on the right cosets $$G/H$$ of $$H\le G$$. Let $$\chi$$ be the character of the permutation representation of $$G$$ defined by this action. Let $$1_G$$ be the trivial character of $$G$$.

Then for $$g\in G$$ we have that $$\chi(g)$$ is the number of points in $$H/G$$ fixed by $$g$$.

But $$g$$ fixes $$Hx$$ if and only if $$G\in H^x$$ so $$\chi(g)$$ is the number of conjugates $$K$$ of $$H$$ with $$g\in K$$.

Denoting by $$Cl(H)$$ the set of subgroups of $$G$$ conjugate to $$H$$ and $$(x,K)=\cases{1 & x\in K \\ 0 & x\notin K}$$ we have
$$\begin{array}{ll} \langle \chi, 1_G\rangle & =\frac{1}{|G|}\sum\limits_{g\in G}\chi(g) \\ & =\frac{1}{|G|}\sum\limits_{g\in G}\sum\limits_{K\in Cl(H)}(g,K) \\ & =\frac{1}{|G|}\sum\limits_{K\in Cl(H)}\sum\limits_{g\in G}(g,K) \\ & =\frac{1}{|G|}\sum\limits_{K\in Cl(H)}|K| \\ & =\frac{1}{|G|}|Cl(H)||H| \\ & =\frac{|H|}{|N_G(H)|} \\ \end{array}$$

This is clearly not always an integer, when $$\langle \chi, 1_G\rangle$$ is. Where is the error?

• It is the normaliser, but it's the other way round - $H$ is a subgroup of $N_G(H)$ so the value is not an integer if $H$ is a proper subgroup of $N_G(H)$ – Robert Chamberlain Feb 3 at 9:34
• I think the error is that you only count conjugates of $H$, whereas the character cares about which element does the conjuating ($H^x = H^y$ does not imply that $Hx = Hy$, only that the cosets for the normalizer are equal, which explains why you get this in the denominator rather than $|H|$ as you should). – Tobias Kildetoft Feb 3 at 10:00
• Thank you @Tobius, write this as an answer and I'll mark it as correct – Robert Chamberlain Feb 3 at 10:02

Your computation looks correct to me, in that $$\chi(g)=|\{x\in H\backslash G|xgx^{-1}\in H\}$$.
This leads one to \begin{aligned} \langle\chi,1\rangle&=\frac1{|G|}\sum_{g\in G}\sum_{x\in H\backslash G}1_{xgx^{-1}\in H}\\ &=\frac1{|G|}\sum_{x\in H\backslash G}|xHx^{-1}|\\ &=\frac1{|G|}\sum_{x\in H\backslash G}|H|\\ &=\frac1{|G|}|H||G|/|H|=1. \end{aligned}
This what we would expect , since writing $$e_{Hx}$$ for the unit basis vector generated by the right coset $$Hx$$ we would get an invariant subspace spanned by $$\sum_{x\in H\backslash G}e_{Hx}$$.
What does wrong in your proof is taking the sum over Cl$$(H)$$, which has size $$|G|/|N_G(H)|$$ as opposed to $$|G|/|H|$$. This is because if you have $$xH\neq x'H$$, but $$xHx^{-1}=x'Hx'^{-1}$$, then you want to make sure to count that twice.