$\DeclareMathOperator{\GL}{GL} \DeclareMathOperator{\End}{End} $Let $\mathbb K$ be a field and $A$ a finite subgroup of $\GL_n\mathbb K$. For each $M \in \GL_n\mathbb K$, let's define a ring endomorphism $\psi_M \in \End \mathbb K[x_1,\dots,x_n]$ such that \begin{equation} \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix} \mapsto M \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix} . \end{equation}
$f \in \mathbb K[x_1,\dots,x_n]$ is called invariant under $A$ if $\psi_M(f) = f$, for all $M \in A$. The problem is to find the ring of the invariants. For example, for $\mathbb K = \mathbb R$, $n=2$ and $A =\langle -I_2\rangle$, the ring of the invariants is $\mathbb R[x^2,y^2,xy]$. A more complex example can be found here. We can do the same with rational functions instead of polynomials, can't we? I'm not sure the term invariant theory is the correct one, though.
Are there any books or lecture notes about this topic?