Why is the sum of irreducible representations nonzero only when the irreducible representation is trivial?

I am going through Normal Subgroup Reconstruction and Quantum Computation Using Group Representations. In section 3, the authors discuss the probability of measuring the irreducible representation $\rho$.

The function is defined in Definition 2, page 629 as follows.

$$f : G \to \mathbb{C}$$

Here $G$ is a finite group. $f$ is the indicator function of a left coset of $H$ in $G$, i.e. for some $c \in G$,

$$f (g) = \begin{cases} \frac{1}{\sqrt{|H|}} & \quad \text{if } g \in c H, \text{and}\\ 0 & \quad \text{otherwise}\\ \end{cases}$$

Then in Lemma 1, it is said that the Fourier transform of $f$, $\hat{f}(\rho)$ is a projection.

The section following the lemma mentions the following points.

The restricted representation from $G$ to $H$, $Res_{H}\rho$ can be decomposed into irreps over $H$. Then the Fourier transform of $f$ at $\rho$ is comprised of blocks, each corresponding to a representation in the decomposition of $Res_H\rho$. The matrix $\sum_{h \in H} \rho(h)$ is as follows.

$$U\begin{bmatrix} \sum_{h\in H} \sigma_1(h) & 0 & \ldots & 0\\ 0 & \sum_{h\in H} \sigma_2(h)& \ldots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \ldots &\sum_{h\in H} \sigma_t(h) \end{bmatrix} U^\dagger$$ for some unitary transformation $U$ and some irreps $\sigma_i$ of $H$. I understand that we need to conjugate it with $U$ for basis change.

Then comes the statement I do not understand.

We know that the sum is nonzero only when the irreducible representation is trivial, in which case it is $|H|$.

Which representation are we talking here? $\rho$ or $\sigma$? Why cannot the sum be nonzero when the representation is not trivial?

Suppose $\sigma: H\to\mathrm{GL}(W)$ is a complex irreducible representation of $H$. Call the trace $\chi$. Then character theory says that $\sum_{h\in H}\chi(h)$ equals $|H|$ if $\sigma$ is trivial and $0$ otherwise. Additionally, the sum $\sum_{h\in H}\sigma(h)$ is an intertwining operator, and Schur's lemma says the intertwining operators of a complex irreducible representation are scalar multiplication maps. Thus, $\sum_{h\in H}\sigma(h)=\lambda\,\mathrm{id}$ for some scalar $\lambda$. Taking traces of both sides, we see $\lambda=|H|$ if $\sigma$ is trivial else is the zero map.

Some background on character theory. If $\chi$ and $\psi$ are irreducible characters (traces of irreducible representations) then they satisfy the following orthogonality relation:

$$\langle \chi,\psi\rangle:=\frac{1}{|H|}\sum_{h\in H}\chi(h)\overline{\psi(h)}=\begin{cases} 1 & \chi=\psi \\ 0 & \chi\ne\psi\end{cases}$$

Then the sum $\sum_{h\in H}\chi(h)$ is a special case with $\psi$ the trace of the trivial representation (which is identically $1$ as a function of $H$).

This answer explains how one can obtain the Peter-Weyl decomposition of finite group algebras and subsequently solve for the orthogonal idempotents and then obtain the fundamental orthogonality relations of character theory using them.

In any case, the orthogonality relations of character theory and Schur's lemma are two of the most basic and powerful tools used in representation theory, and anyone working in representation theory would benefit from learning them.

• When you said $\chi$ and $\Psi$ are irreducible characters, did you mean that they are the traces of the two matrices which the representations of the same group element for two different irreps? – Omar Shehab May 2 '16 at 19:39
• What you said is not actually a sentence. Keep in mind a representation is not a matrix, it is a matrix-valued function of group elements. Taking traces is a scalar-valued function of matrices. Composing the two gives a scalar-valued function of group elements, which is called the character of the representation. That is, if $g\mapsto \sigma(g)$ is a representation, then $g\mapsto{\rm tr}\,\sigma(g)$ is a character. The irreducible characters are those that come from irreducible representations. All of this is standard terminology. – arctic tern May 2 '16 at 19:51