Representations of an algebraic group $G$ versus representations of the group $G(k)$ The following question stems from the discussion here.
Let $G$ be a group scheme, $G$ acting on a vector space $V$ over a field $k$ is morphism of group valued functors $G \rightarrow GL_V$, where $GL_V$ is the functor on commutative $k$-algebras defined by:
$$
R \mapsto \text{Aut}_{R-\text{Mod}}(V\otimes_k R)
$$
This is I believe a standard definition and is used by J.S. Milne in his book on algebraic groups.
Consider the case when $G$ is actually a variety, over an algebraically closed field $k$, say an affine or quasi-projective one. Then from a geometry perspective I can think of $G$ as being it's set of $k$ points like chapter 1 of Hartshorne and $G(k)$ is just an abstract group, but when $G$ has coordinates then $G(k)$ is defined by polynomials and the group operations are given by regular functions. Lets say $G(k)$ acts on some vector space. When is this equivalent to the scheme theoretic notion above?
Example when these notions don't allign even though they might seem to is the following. Any $G$ rep in the scheme theoretic sense is an increasing union of finite dimensional reps, it's "rational", this is corollary 4.7, page 71, of in J.S. Milne's notes linked above. But consider the case of $(\mathbb{C},+)$, (the $\mathbb{C}$ points of $\mathbb{G}_a$) acting  on $\mathbb{C}(t)$. Where $g\cdot f(t)=f(t+g)$. Then the $G$-submodule spanned by $1/t$ is infinite dimensional. Meaning that this representation is not something that comes out of a $\mathbb{C}[t]$-codmodule structure like group scheme theoretic rep of $\mathbb{G}_a$ over $\mathbb{C}$ would.
So given a rep of $G(k)$ how do I know it's the same as the $k$-points of some a scheme theoretic rep, $G\rightarrow GL_V$
I'm especially interested in the case of an affine algebraic group $G$ over an algebraically closed field $k$ acting on a variety and thus $G(k)$ acting it's field of rational functions $k(G)$.
Edit: Note that in the finite dimensional case if $G$ is quasi-projective and $k$ is algebraically closed then $G(k)$-reps as an abstract group which are defined by polynomial equations are the same as scheme theoretic maps $G\rightarrow GL_n$, because maps of varieties in the sense of Hartshorne chapter 1 are equivalent to schemes maps. So in particular this question is of interest when the representation is infinite dimensional.
Edit: The lack of the word "acting" in $G(k)$ acting on it's field of rational functions $k(G)$ was corrected. I also added a definition for $GL_V$ when $V$ is infinite dimensional.
 A: This is more than anything a complement/continuation of hm2020's answer, especially the second part denoted by Answer in hm2020's post.
The point of how to think of $\mathrm{GL}(V)$ for infinite-dimensional $V$ is kind of subtle, because already the question of how to think of a "usual" $k$-vector space $V$ as some kind of $k$-scheme itself becomes subtle.
The two flavors of "geometric" vector spaces
There are two common ways of associating a $k$-scheme to a $k$-vector space $V$:

*

*$X_V = \mathrm{Spec}(\mathrm{Sym}^*_k(V^\vee))$

*$Y_V = \mathrm{Spec}(\mathrm{Sym}^*_k(V))$
For instance, (if I am not mistaken,) Hartshorne and the EGA use 1, while Fulton's Intersection Theory uses 2 (all of them in the more general context of vector bundles, i.e. over a base scheme that isn't necessarily $\mathrm{Spec}(k)$). In either case, once the convention is understood, the "geometric vector space" would usually just be denoted $V$. But we are trying to draw attention to the distinction between them, hence the awkward notion.
The advantage of 1 is that, for a commutative $k$-algebra $A$, the $A$-points of the "geometric" vector space associated to $V$ are $$X_V(A) = \mathrm{Hom}_k(V^\vee, A).$$When $V$ is finite-dimensional, this is further equal to $X_V(A) = V\otimes_k A,$ which is surely what one wants to do from a "functor of points" perspective!
The disadvantage of 1 is that $X_V$ only knows about the $k$-linear dual $V^\vee$, and not of the vector space $V$ itself. Indeed, if you define the notion of a linear map of "geometric" $k$-vector space schemes correctly (doable but annoying), you will discover that $$\mathrm{Hom}_{\mathrm{Vect}(\mathrm{Sch}_k)}(X_V, X_{V'})\simeq \mathrm{Hom}_k(V'^\vee, V^\vee).$$When $V$ and $V'$ are finite-dimensional, this is equal to $\mathrm{Hom}_k(V, V')$, but not otherwise. Hence for non-finite-dimensional vector spaces, the embedding $V\mapsto X_V$ of "algebraic" into "geometric" vector spaces is not fully faithful.
This is where option 2 above is better. With it, we will have
$$\mathrm{Hom}_{\mathrm{Vect}(\mathrm{Sch}_k)}(Y_V, Y_{V'})\simeq \mathrm{Hom}_k(V', V)$$
which, while contravariant, is at least fully faithful. But now "algberaic = usual" $k$-vector spaces, and "geometric" $k$-vector space schemes are anti-equivalent, not equivalent categories as we might desire!
Also, the set of $A$-points of the $k$-scheme associated to $V$ under 2 is
$$
Y_V(A) = \mathrm{Hom}_k(V, A)
$$
which will never be (canonically) equal to $V\otimes_k A$. For finite-dimensional vector spaces $V$, you will get it to be $V^\vee\otimes_k A$, but for infinite-dimensional ones not even that.
Back to the example in question & take-away
In the example from the original question, one might imagine that the $\mathbf C$-vector space $V=\mathbf C(t)$ might be the set of $\mathbf C$-points of some natural associated $\mathbf C$-scheme. But as the above discussion tried to demonstrate, it is not clear how to do so. Our best bet would be to use 2 and get the $\mathbf C$-scheme $Y_{\mathbf C(t)}$ with $\mathbf C$-points $$Y_{\mathbf C(t)}(\mathbf C) = \mathrm{Hom}_{\mathbf C}(\mathbf C(t), \mathbf C)= \mathbf C(t)^\vee.$$
Thus I think that when people make assertions such as "The category of representations of a group scheme $G$ is equivalent to the category of comodules of the Hopf algebra $\mathcal O(G)$", they must probably have either

*

*only have the finite-dimensional case in mind, in which case there are no issues, or

*allow for arbitrary dimensionality of representations, but suppress the fact that there is really an anti-equivalence, with the arrows in the "geometric" representations going the other way than in the comodules.

In particular, in the regime 2 (which is, so far as I understand, the only one possible if we want to speak about infinite-dimensional reps), a $G$-rep will be equivalently

*

*A $k$-vector space $V$, together with a $G$-action
$G\times_{\mathrm{Spec}(k)}Y_V\to Y_V$ on the associated "geometric"
vector space $Y_G$.

*A $\mathcal O(G)$-comodule structure on $V$.

In the former description, you can try to get some mileage by thinking of $Y_V$ as cut out by polynomial equations inside some very big affine space, if you so wish. But on the level of $k$-points, you will get a map $$G(k)\times \mathrm{Hom}_k(V, k)\to \mathrm{Hom}_k(V, k)$$or, if you prefer to write it in terms of dual vector spaces,$$G(k)\times V^\vee\to V^\vee.$$
If you wish, you can try to find a vector space $V$ such that there is a natural identification $V^\vee =\mathbf C(t)$, but I don't know how to formulate an action that might look like the $f(t)\mapsto f(t+a)$ for $a\in k=\mathbf G_a(k)$ that was asked about in the original question.

Edit: The edit to the original post clarified that we are following Milne's "Algebraic Groups" book.
Vector spaces as functors, or, the secret third flavor
For a (usual = algebraic) $k$-vector space $V$, we thus define the functor $GL_V:\mathrm{CAlg}_R\to\mathrm{Grp}$ by $\mathrm{GL}_V(R) := \mathrm{Aut}_R(V\otimes_k R)$. This is equivalent to adopting a third perspective on how to incarnate a (possibly infinite-dimensional) vector space $V$ as an object of algebraic geometry:


*As the functor $V_\mathfrak{a}:\mathrm{CAlg}_k\to\mathrm{Vect}_k$,given by $V_\mathfrak{a}(R):= V\otimes_k R$.

Then $\mathrm{GL}_V$ is quite literally automorphisms of $V_\mathfrak{a}$, and a map $G\to\mathrm{GL}_V$ in $\mathrm{Fun}(\mathrm{CAlg}_k, \mathrm{Grp})$ is equivalent to a map of functors $G\times V_\mathfrak{a}\to V_\mathfrak{a}$, satisfying all the usual requirements.
But that is a highly functor-of-points perspective, that is for infinite-dimensional vector spaces $V$ increasingly far from varieties. Indeed, $V_\mathfrak{a}$ is not an algebraic variety if $\dim(V) =\infty$. It might or might not be representable by a scheme (I don't know), but if it is, it isn't by the expected usual friendly "geometric vector space" scheme $X_V=\mathrm{Spec}(\mathrm{Sym}^*_k(V^\vee))$.
What is contained in a representation
But with this definition, it isn't too hard to explain (though satisfaction may vary) what extra data is encoded in saying "$V$ is a $G$-rep". Namely, the map $\rho_k:G(k)\times V\to V$ is required to extend functorially to $R$-linear maps
$$\rho_R : G(R)\times (V\otimes_k R)\to V\otimes_k R$$
for all commutative $k$-algebras $R$.
The (key) functoriality requirement means that, for any $k$-algebra map $\varphi:R\to R'$, the corresponding $R'$-linear map
$$
G(R)\times (V\otimes_k R)\xrightarrow{G(\varphi)\times (\mathrm{id}_V\otimes_k \varphi)}G(R')\times (V\otimes_k R')\xrightarrow{\rho_{R'}} V\otimes_k R'
$$
agrees with
$$
G(R)\times (V\otimes_k R)\xrightarrow{\rho_R} V\otimes_k R\xrightarrow{\mathrm{id}_V\otimes_k \varphi} V\otimes_k R'.
$$
The 'counterexample' in question
Let us try to make the counterexample from the original post work in this context! Let us first pursue this naively, leading to a contradiction, and only afterward point out the issue.
We have $k=\mathbf C$ (though this shouldn't matter), $V=\mathbf C(t)$, and $G=\mathbf G_a$. Thus for any $R\in\mathrm{CAlg}_k$ we have
$$
V_\mathfrak{a}(R) = R(t), \quad \mathbf G_a(R) = R
$$
and the action $\rho_R: R\times R(t)\to R(t)$ should presumably be defined as
$$
\rho_R(a, f(t)) = f(t+a)
$$
for a scalar $a\in R$ and rational function $f(t)\in R(t)$. This clearly is a group homomorphism, and the functoriality condition boils down to the manifestly true requirement that wd have for any $k$-algebra map $\varphi: R\to R'$, $a\in R$, and $f(t)\in R(t)$
$$\varphi(f(t+a)) = \varphi(f)(t+\varphi(a)).$$
This would imply that we have a legitimate representation on our hands, hence that it corresponds to a comodule, and consequently that it is rational (every element is contained inside a finite-dimensional subrep), which we know to be violated, as explained in the original question.
So where did we go wrong?
Answer: in asserting that $V_{\mathfrak{a}}(R) = R(t)$. This sure feels right, and would certainly hold with polynomials in place of rational functions, but isn't true here. Indeed, we instead have by definition $V_\mathfrak{a}(R) = \mathbf C(t)\otimes_{\mathbf C}R$, and as is elegantly demonstrated here, this tensor product is instead
$$
\mathbf C(t)\otimes_{\mathbf C}R = (\mathbf C[t]-0)^{-1}R[t].
$$
That is to say, we are only allowed to divide by polynomials with coefficients in the base field $\mathbf C$.
This seriously hinders the above-sketched program for defining the representation $\rho$ by $\rho_R(a, f(t)) = f(t+a)$ for $a\in R$ and $f\in V_{\mathfrak{a}}(R)$. Indeed, if we write this out, we have $f(t)=\frac{p(t)}{q(t)}$ for $p(t)\in R[t]$ and $q(t)\in\mathbf C[t]-0$. An element $a\in R=\mathbf G_a(R)$ would therefore want to act by
$$
f(t)=\frac{p(t)}{q(t)}\mapsto f(t+a) = \frac{p(t+a)}{q(t+a)},
$$
which is naturally a problem on account of the polynomial $q(t+a)$ now having coefficients in $R$, and so it is unclear if dividing by it is even allowed in $\mathbf C(t)\otimes_{\mathbf C}R$.
In short: it is unclear how one would go about extending the $\mathbf C$-point level representation $\mathbf C\times \mathbf C(t)\to\mathbf C(t)$ to a representation on the level of functors.
A: Question: "Let G be a group scheme, G acting on a vector space V is morphism of grouph schemes $G→GL(V)$. Consider the case when G is actually a variety, over an algebraically closed field k, say an affine or quasi-projective one. Then from a geometry perspective I can think of G as being it's set of k points like chapter 1 of Hartshorne and $G(k)$ is just an abstract group. Lets say $G(k)$ acts on some vector space. When is this equivalent to the scheme theoretic notion above?"
Answer: If $G(k):=V(\mathfrak{p})$ where $A:=k[y_1,..,y_m]/\mathfrak{p}$ with $\mathfrak{p}$  a prime ideal and $GL(V)(k):=V(tdet(x_{ij})-1)$ with $B:=k[x_{ij},t]/(tdet(x_{ij})-1)$ it follows $G(k), GL(V)(k)$ are affine algebraic varieties of finite type over $k$ in the sense of Hartshorne, Chapter I. Def 1.1.4. Let $G:=Spec(A), GL(V):=Spec(B)$.
By HH.Prop.I.3.5 and HH.Prop.II.2.3 it follows
$$Hom_{var}(G(k), GL(V)(k))\cong Hom_{k-alg}(\mathcal{O}(GL(V)(k)), \mathcal{O}(G(k))) \cong$$
$$ Hom_{k-alg}(B,A) \cong Hom_{Sch}(G, GL(V)).$$
A $G(k)$-module $\rho^*: G(k) \rightarrow GL(V)(k)$ is in particular a map of algebraic varieties, hence there is a map of $k$-algebras $\rho: B \rightarrow A$ inducing $\rho^*$. The set $Hom_{var}(G(k), GL(V)(k))$ is the set of maps of varieties over $k$ in the sense of HH.CH.I. The set $Hom_{Sch}(G, GL(V))$ is the set of maps of affine schemes over $k$.
This is a 1-1 correspondence when $G(k)$ is affine.
In HH.Ex.II.2.15 they prove that for any algebraically closed field $k$ and any varieties $V(k), W(k)$ (in the sense of Ch I) there is a 1-1 correspondence
$$Hom_{Var}(V(k), W(k)) \cong Hom_{Sch}(t(V(k)), t(W(k))).$$
Hence if $G(k)$  is quasi projective over $k$ it follows
$$Hom_{Var}(G(k), GL(V)(k)) \cong Hom_{Sch}(t(G(k)), t(GL(V)(k))).$$
Question: "So in particular this question is of primary interest when the representation is infinite dimensional."
Answer: You should include a definition of $GL_k(V):=Spec(A)$ when $V$ is an infinite dimensional $k$-vector space - what is $A$? You may of course consider the "set of  $k$-linear automorphisms" of $V$ but in your question you are asking about "maps of group schemes" and then you must define $GL_k(V)$ as a "group scheme".
Example: If $k$ is a field and $V$ is a $k$-vector space, you may consider the group $GL_k(V)$ of $k$-linear automorphisms of $V$. If $V$ is finite dimensional we define $GL_k(V):=Spec(A)$ where $A:=k[x_{ij},t]/(tdet(x_{ij})−1)$, and $GL_k(V)$ is an affine group scheme. What is your definition of $A$ when $V$ is infinite dimensional?
The problem is that if $\phi: V \rightarrow V$ is a $k$-linear automorphism, it is not clear how to define $det(\phi)$. You cannot simply define $A:=k[x_{i,j}: i,j \in \mathbb{N}]$ and let $B:=A[t]/(t det(x_{ij})-1)$ with $GL_k(V):=Spec(B)$ since there is no algebraic definition of a "determinant" for "infinite matrices" such as $(x_{ij})$. Hence you must specify: What is $GL_k(V)$ for infinite dimensional $V$?
Question: "I'm especially interested in the case of an affine algebraic group G over an algebraically closed field k acting on a variety and thus $G(k)$ it's field of rational functions $k(G)$."
Answer: If $G:=Spec(A)$ where $A:=k[x_1,..,x_n]/\mathfrak{p}$ is a finitely generated $k$-algebra with $\mathfrak{p}$ a prime ideal, it follows
$$G(k):=V(\mathfrak{p}) \subseteq \mathbb{A}^n_k.$$
Note that $G(k)$ is by definition a set and not a field.
