This might be a stupid question, but I could not find a good reference that thoroughly explains the matter. I will start with some lengthy introduction.
If $\mathcal{A}$ is any abelian category, then we have functors $\mathrm{Hom}_\mathcal{A} (M,-)$ and $\mathrm{Hom}_\mathcal{A} (-,N)$, both left exact, covariant and contravariant, so if we have enough injectives (resp. projectives), we can calculate their right derived functors $\mathrm{Ext}^n_\mathcal{A} (M,-)$ and $\mathrm{Ext}^n_\mathcal{A} (-,N)$.
Now if $\mathcal{A}$ is the category or $R$-modules, things become more interesting: $\underline{\mathrm{Hom}}_R (-,-)$ has also an $R$-module structure, so it is not merely a functor with values in abelian groups, and $\mathrm{Ext}^n_R (M,N)$ has an $R$-module structure. Of course, the forgetful functor $R\mathcal{-Mod} \to \mathcal{Ab}$ is exact, so if we are not interested in the module structure, we can just derive the $\mathrm{Hom} (-,-)$ with values in $\mathcal{Ab}$. I think many introductory texts or classes ignore the $R$-module structure.
Things start to confuse me when $R$ is non-commutative. We have to distinguish left, right modules, and bimodules. So which $\mathrm{Hom} (-,-)$ functor we should derive (defined on which categories of modules and bimodules and with values in which category of modules/bimodules)? For instance, I think the Yoneda Ext naturally has a bi-module structure (for extensions we take pullbacks or pushouts over the action maps $r\colon M\to M$ and $r\colon N\to N$).
And I have the same question for tensor products and $\mathrm{Tor}$ functors for modules over noncommutative rings.
Edit: I actually wanted to see some reference that, for example, treats $\mathrm{Ext}$ as a derived functor of something like $\underline{\mathrm{Hom}}_S (-,-)\colon (R,S)\text{-bimod}^\mathrm{op} \times (T,S)\text{-bimod} \to (T,R)\text{-bimod}$ but this might be some ridiculous generality.