Sum of certain integer numbers related to the elements of a finite abelian group and to its group of characters We consider a finite abelian group $G$ and its group characters $G^*$. For each $g\in G$ and $\chi\in G^*$ we define $0\leq r_g^\chi< o(g)-1$ such that $\chi(g):=e^{\frac{2\pi i}{o(g)}r_g^\chi}$.
We fix two elements $g,h\in G$. I want to find a closed formula for the following sum:
$\sum_{\chi \in G^*}r_g^\chi r_h^\chi$
I found already the closed form for the following sum:
$\sum_{\chi \in G^*}r_g^\chi=\frac{|G|}{2}(o(g)-1)$
but I don't know how to get the other closed form.
A little remark can be the following
$\frac{|G|}{2}(o(g)-1)\frac{|G|}{2}(o(h)-1)=(\sum_{\chi\in G^*}r_g^\chi )(\sum_{\eta\in G^* }r_h^\eta)=\sum_{\chi,\eta \in G^*}r_g^\chi r_h^\eta= (\sum_{\chi \in G^*}r_g^\chi r_h^\chi)+(\sum_{\chi\neq \eta  \in G^*}r_g^\chi r_h^\eta)$
and so
$\sum_{\chi \in G^*}r_g^\chi r_h^\chi= \frac{|G|}{2}(o(g)-1)\frac{|G|}{2}(o(h)-1)- \sum_{\chi\neq \eta  \in G^*}r_g^\chi r_h^\eta$
 A: Here is a partial answer. Let $f(g,h)=\sum_{\mathbb\chi\in G^*}r_g^\chi r_h^\chi$ be the required sum for $g,h\in G.$ I can express this in a similar form to your expression for $\sum r^\chi_g,$ but much more complicated, see equation (1). I'm not sure if there is a ``nice'' expression.
Let $g$ have order $m$ and $h$ have order $n$ and let $|K|=d$ where $K=\langle g\rangle\cap\langle h\rangle.$ Let $H=\langle g,h\rangle,$ so $|H|=mn/d.$ Since $g^{m/d}$ and $h^{n/d}$ both generate $K,$ there is unique $u\in(\mathbb Z/d\mathbb Z)^\times$ such that $f^{n/d}=(g^{m/d})^u.$
Let $\zeta_m=e^{2\pi i/m}$ and $\zeta_n=e^{2\pi i/n}.$ Then $ H^*=\{\lambda_{a,b}:au=b\text{ mod }d\},$ where $\lambda_{a,b}:H\mapsto\mathbb C^\times$ is defined by $\lambda_{a,b}(g)=\zeta_m^a$ and $\lambda_{a,b}(h)=\zeta_n^b.$
To see this note every $\lambda\in H^*$ i.e. every homomorphism $H\rightarrow\mathbb C$ is of the form $\lambda_{a,b}$ for some $a,b,$ because $g$ and $h$ must be mapped to powers of $\zeta_m$ and $\zeta_n$ and the congruence condition ensures $f^{-n/d}(g^{m/d})^u$ maps to $1.$ Also $a$ and $b$ are unique mod $m$ and $n$ respectively. But the allowed $a,b$ are just those in the kernel of the map $\mathbb Z/m\mathbb Z\oplus \mathbb Z/n\mathbb Z\rightarrow\mathbb Z/d\mathbb Z,$ $(a,b)\mapsto au-b,$ so there are $mn/d=|H|$ such $(a,b),$ as required.
As $r^{\lambda_{a,b}}_g=a$ and $r^{\lambda_{a,b}}_h=b,$ and each $\lambda_{a,b}$ has $|G:H|$ extensions to $G,$ we get
\begin{equation}\tag{1}
f(g,h)=|G:H|\sum_{a=0}^{m-1}\sum_{b=0,b=au\text{ mod }d}^{n-1}ab.
\end{equation}
I will sketch some further evaluation of the RHS but it seems to be messy. Write $m=dm_1,n=dn_1,$ and $a=rd+a_1,$ $b=sd+b_1$ where $0\le r<m_1,$ $0\le s<n_1,$ $0\le a_1,b_1<d.$ Then
$$|G:H|^{-1}f(g,h)=\sum_{r=0}^{m_1-1}\sum_{s=0}^{n_1-1}\sum_{a_1=0}^{d-1}(rd+a_1)(sd+\overline{a_1u})$$ where $\overline{a_1u}$ means the remainder of $a_1u$ mod $d.$ The RHS expands to $$d^3\sum_{0}^{m_1-1} r\sum_0^{n_1-1}s+m_1d\sum_0^{d-1}a_1\sum_0^{n_1-1}s+n_1d\sum_0^{m_1-1}r\sum_{a_1=0}^{d-1}\overline{a_1u}+m_1n_1\sum_{a_1=0}^{d-1} a_1\overline{a_1u}.$$ The first three summands are straightforward (note the second factor in the third summand is just $d(d-1)/2$ because $u$ is prime to $d$) and I won't write them down. For the last term I find (verified by excel):
$$\sum_{a_1=0}^{d-1} a_1\overline{a_1u}=d\left(\frac 12\sum_{j=1}^u\left(\left[\frac{jd}u\right]\left(\left[\frac{jd}u\right]-1\right)\right)-\frac{d-1}6(u(d-2)+3)\right).$$
In principle then we have an expression for the RHS of (1) which is polynomial in $m,n,d,u$ apart from the sum over $j$ above - I'm not sure if this can be improved or interpreted in a nice way.
