Schur orthogonality relation between equivalent but unequal representations?

Question

The Schur orthogonality relation in coordinate/matrix element form reads $$\sum_{R\in G}^{|G|} \; \Gamma^{(\lambda)} (R)_{nm}^*\;\Gamma^{(\mu)} (R)_{n'm'} = \delta_{\lambda\mu} \delta_{nn'}\delta_{mm'} \frac{|G|}{l_\lambda}$$ where $\lambda,\mu$ lable the irreducible representations of group $G$. But there seems to be a case not specified clearly.

What happens when the two representations $\lambda$ and $\mu=\lambda'$ are equivalent but not identical to each other, i.e., related by a similarity transformation. What the relation would be in this case?

This is not an issue in the orthogonality relation in character form since Tr() is invariant under a similarity transformation. But for the matrix form above, the proof seems to be affected. For instance, when finding the normalization factor, one proof I saw uses (setting $\mu=\lambda$) $$c_{m'm}l_\lambda=\sum_{R}\sum_n \; \Gamma^{(\lambda)} (R^{-1})_{mn}\;\Gamma^{(\lambda)} (R)_{nm'} = \sum_{R} \; \Gamma^{(\lambda)} (R^{-1}R)_{mm'}.$$ But how about setting $\mu=\lambda'$? I feel it is not immediately true to have the second equality since $\Gamma^{(\lambda)}$ and $\Gamma^{(\lambda')}$ are not identical.

Answer 1

Suppose $\langle-,-\rangle$ is a complex inner rep on a $G$-irrep $U$. If $u,v\in U$ are two vectors, we say $g\mapsto\langle u,gv\rangle$, as a function $G\to\mathbb{C}$, is a matrix coefficient. If $u,v$ are basis vectors in some ordered basis, this is literally the corresponding entry of $g$ represented as a matrix.

The functions $G\to\mathbb{C}$ form their own complex inner product space $\mathbb{C}^G$ with inner product $\langle f_1,f_2\rangle=\frac{1}{|G|}\sum_{g\in G}\overline{f_1(g)}f_2(g)$. Schur orthogonality says

$$ \big\langle \langle a,gb\rangle,\langle c,gd\rangle \big\rangle=\langle c,a\rangle\langle b,d\rangle/\dim U. $$

An equivalent representation on $U$ uses $TgT^{-1}$ instead of $g$, for some $T\in GL(U)$. Observe

$$ \big\langle\langle a,TgT^{-1}b\rangle,\langle c,gd\rangle\big\rangle =\langle Tc,a\rangle\langle T^{-1}b,d\rangle/\dim U. $$

If we pick $a,b,c,d$ from an orthonormal basis for $U$, the above values are not necessarily $0$ or $1$ over $\dim U$; as we let $a,b,c,d$ be orthonormal and let $T$ vary, the right side can be many kinds of values. This means there's very little to no relation between matrix coefficients of different but equivalent irreps like we see with Schur orthogonality for a given irrep.