How is this a group action on the polynomial ring? (Eisenbud Commutative Algebra page 29) I have asked a question about the first part of the paragraph in question here, and after thinking about it for a while I determined that the resulting action was indeed a group action. But I came back to this for use in another example, and I am confused again.
The action is by $GL_n(k)$ ($k$ a field) on the polynomial ring $S=k[x_1,\dots,x_n]$, and it is given by $g(f)(x_1,\dots,x_n)=f(g^{-1}(x_1),\dots,g^{-1}(x_n))$, where we view $GL_n(k)$ as the general linear group of span$\{x_1,\dots,x_n\}$.
The problem is, I seem to get contradictory results depending on which order I apply the group action in the case $g(h(f))$. If I interpret $g(h(f))$ as $g$ acting on $h(f)$, then I get $$g(h(f))(x_1,\dots,x_n)=(h(f))(g^{-1}(x_1),\dots,g^{-1}(x_n))=f(h^{-1}g^{-1}(x_1),\dots,h^{-1}g^{-1}(x_n)),$$ where the last equality is from taking $h(f)$ but first replacing the $x_i$'s in $f$ by $g^{-1}(x_i)$. This is the correct result.
I tried to go about this by having $h$ act on $f$ first, and then have $g$ act on that polynomial, but this does not appear to give the correct result. Here is an example of it failing:
Let $S=k[x_1,x_2]$ and $f=x_1$. Also let $g^{-1}$ and $h^{-1}$ be represented by $(a_{ij})$ and $(b_{ij})$ respectively. Then $h(f)(x_1,x_2)=h^{-1}(x_1)=b_{11}x_1+b_{21}x_2$. Then shouldn't $g$ act on $h(f)=b_{11}x_1+b_{21}x_2$ by replacing each $x_i$ by $g^{-1}(x_i)$ (this is the definition...)? Then we get $g(h(f))=b_{11}(a_{11}x_1+a_{21}x_2)+b_{21}(a_{12}x_1+a_{22}x_2)$, which is seen to be the opposite of what would happen if we interpreted this as above.
So, what am I doing wrong when I evaluate it like that? Also, is there a way to evaluate it "outwards" like I tried to do, but actually achieve the desired result?
I reckon that somehow the notation $g(h(f))(x_1,\dots,x_n)$ as opposed to $g(h(f)(x_1,\dots,x_n))$ has something to do with it, but I am not sure.
 A: The problem is that the notation $g^{-1}(x_i)$ looks unambiguous but it's actually not. There are two things you might interpret this to mean and they give two different results.
As usual, things are cleaner if we first don't allow ourselves to work in coordinates. Let $V$ be a finite-dimensional vector space over $k$. There are two polynomial rings you can construct out of $V$, and $GL(V)$ acts on both of them:

*

*the symmetric algebra $S(V) = \sum_{n \ge 0} S^n(V)$, where $S^n(V)$ is the quotient of the tensor power $V^{\otimes n}$ by the action of $S_n$. The action of $GL(V)$ on $S(V)$ extends the tautological action of $GL(V)$ on $V$ by addition and multiplication.

*the symmetric algebra $S(V^{\ast}) = \sum_{n \ge 0} S^n(V^{\ast})$, whose elements can be interpreted as polynomial functions on $V$ ($S(V)$ is then polynomial functions on $V^{\ast})$. The action of $GL(V)$ on $S(V^{\ast})$ extends the dual action of $GL(V)$ on $V^{\ast}$.

Now we pick a basis $v_1, \dots v_n \in V$. This induces a dual basis $f_1, \dots f_n \in V^{\ast}$ defined by $f_i(v_j) = \delta_{ij}$, and these bases allow us to identify our symmetric algebras as polynomial algebras
$$S(V) \cong k[v_1, \dots v_n]$$
$$S(V^{\ast}) \cong k[f_1, \dots f_n].$$
If we now use the basis $v_i$ to identify $GL(V)$ with $GL_n(k)$, then $g \in GL_n(k)$ acts on a sum $\sum c_i v_i \in V$ via matrix multiplication of $g$ on the column vector with entries $c_i$ in the usual way. It also acts on a sum $\sum r_i f_i \in V^{\ast}$, but now via multiplication by the matrix $(g^T)^{-1}$ (note the transpose); said another way, it acts via the action of $g^{-1}$ on the row vector with entries $r_i$.
So, explicitly, if $g^{-1} = \left[ \begin{array}{cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array} \right]$, then there are two ways to interpret the meaning of $g^{-1}(x_1)$, depending on whether you're thinking of $x_1$ as $v_1$ or as $f_1$:

*

*As $v_1$: then we identify $x_1$ with the column vector $\left[ \begin{array}{c} 1 \\ 0 \end{array} \right]$, which gives $g^{-1}(x_1) = a_{11} x_1 + a_{21} x_2$.

*As $f_1$: then we identify $x_1$ with the row vector $\left[ \begin{array}{cc} 1 & 0 \end{array} \right]$, which gives $g^{-1}(x_1) = a_{11} x_1 + a_{12} x_2$.

If you choose the covariant interpretation ($x_1 = v_1$), which is implied by "general linear group of $\text{span}(x_1, \dots x_n)$," then you don't need to put an inverse in the definition of the action, but you do need to be careful about how you define it. Safest to say the coordinate-invariant thing.
