Residue of Rankin-Selberg L-function for non-trivial nebentypus Let $f\in S_k(\Gamma_0(N),\chi)$ be a normalized holomorphic newform (i.e. weight $k$, level $N$, nebentypus $\chi$) and write its Fourier expansion as
$$ f(z)=\sum_{n\ge 1} \lambda_f(n)n^{(k-1)/2}e^{2\pi i n z}$$
for $\Im z >0$ where $\lambda_f(1)=1$ and, by Deligne's bound, $|\lambda_f(n)|\le d(n)$. Here $d(n)$ denotes the number of positive divisors of $n$. Define the Rankin-Selberg convolution $L$-function as
$$
L(s,f\times \bar{f}) = \sum_{n=1}^\infty \frac{|\lambda_f(n)|^2}{n^s}
$$
for $\Re s >1$. This $L$-function has a simple pole at $s=1$.
Questions: 
(1) What is this residue? 
(2) How does one compute this residue?
 A: As you have said, let $f$ we a weight $k$ form of nebentypus $\chi$. Then in particular, $f(\gamma z) = (cz + d)^k \chi(d) f(z)$, where $\gamma = \left(\begin{smallmatrix} a&b\\c&d \end{smallmatrix}\right)$ is a matrix in our congruence subgroup du jour. 
Notice that $\lvert f(z) \rvert^2 y^k$ is invariant under the slash operator, as
$$ \begin{align}
\lvert f(\gamma z) \rvert^2 (\gamma y)^k &= f(\gamma z) \overline{f(\gamma z)} \frac{y^k}{\lvert cz + d \rvert^{2k}} \\
&= (cz+d)^k \overline{(cz + d)^k} \chi(d) \overline{\chi(d)} f(z) \overline{f(z)} \frac{y^k}{\lvert cz + d\rvert^{2k}} \\
&= \lvert f(z) \rvert^2 y^k.
\end{align}$$
This means that it is meaningful to take the inner product against the normal Eisenstein series
$$ E(z,s) = \sum_{\gamma \in \Gamma_\infty \backslash \Gamma_0(N)} \text{Im}(\gamma z)^s.$$
Performing an unfolding of the integral along the critical strip, we can see that
$$ \langle \lvert f \rvert^2 y^k, E(z,s) \rangle = \frac{\Gamma(s + k - 1)}{(4 \pi)^{s + k - 1}} \sum_n \frac{a(n)^2}{n^{s + k - 1}},$$
where $a(n) = \lambda(n)n^{(k-1)/2}$ are the Fourier coefficients of $f$. Notice the sum on the right is the Rankin-Selberg $L$-function, and I deliberately avoid indicating any normalization anywhere.
As we understand the analytic behaviour of the Eisenstein series and the $\Gamma$ function, we can understand the analytic behavior of the $L$-function. Most importantly, since the Eisenstein series has a single pole at $s = 1$ of known residue, we understand the pole of the $L$-function, and it has residue
$$ \langle fy^{k/2}, fy^{k/2} \rangle R \frac{(4\pi)^k}{\Gamma(k)},$$
where $R$ is the residue of the Eisenstein series attached to the congruence subgroup (and often looks something like $\frac{3}{\pi}$).
A: The answer is that, with your notation,
$$\operatorname{res}_{s=1} L(s,f\times\overline f)
=\lim_{x\rightarrow\infty}\frac1x\sum_{n\leq x}\left|\lambda_f(n)\right|^2.$$
This fact may be found in a paper by Micah Milinovich and Nathan Ng on the arXiv at arXiv:1306.0854 [math.NT]. (See the remark on page 4.)
In the simple case that $f$ is a normalized primitive holomorphic cusp form on the full modular group, then $f$ is self dual and
$$L(s,f\times\overline f)=L(s,f\times f) = \sum_1^\infty \lambda_f^2(n)n^{-s}.$$
Then
$$\operatorname{res}_{s=1} L(s,f\times f)=Z(1,f)M/\varphi(M)$$
(equation (2.35) in `Low-lying zeros of families of $L$-functions' by Iwaniec-Luo-Sarnak [note they use the notation $L(s,f\otimes f)$ for what we describe as $L(s,f\times f)$).
