For any set $S$ and group $G$, the set $\operatorname{Maps}(S,G):=\{f\mid f\colon S\to G\}$, endowed with the pointwise multiplication "$*$":
$$(f*g)(x):=f(x)g(x)\tag 1$$
is a group. In fact:
- closure: for every $f,g\in\operatorname{Maps}(S,G)$, $f*g\in\operatorname{Maps}(S,G)$ by definition $(1)$;
- associativity: for every $x\in S$, $(f*(g*h))(x)=$ $f(x)((g*h)(x))=$ $f(x)(g(x)h(x))=$ $(f(x)g(x))h(x)=$ $((f*g)(x))h(x)=$ $((f*g)*h)(x)$, whence $f*(g*h)=(f*g)*h$;
- identity: $\exists e\in \operatorname{Maps}(S,G)$, defined by $e(x)=1_G$ for every $x\in S$; then, for every $f\in\operatorname{Maps}(S,G)$ and $x\in S$, $(f*e)(x)=$ $f(x)e(x)=$ $f(x)1_G=$ $f(x)$, whence $f*e=f$ for every $f\in\operatorname{Maps}(S,G)$ and hence $e$ plays as identity of $\operatorname{Maps}(S,G)$;
- inverse element: for every $f\in\operatorname{Maps}(S,G)$, $\exists \tilde f\in \operatorname{Maps}(S,G)$, defined by $\tilde f(x)=f(x)^{-1}$ for every $x\in S$; then, for every $f\in\operatorname{Maps}(S,G)$ and $x\in S$, $(f*\tilde f)(x)=$ $f(x)\tilde f(x)=$ $f(x)f(x)^{-1}=$ $1_G$, whence $f*\tilde f=e$ for every $f\in\operatorname{Maps}(S,G)$ and hence $\tilde f$ plays as inverse of $f$.
Now, the subset ${\rm FS}\subseteq\operatorname{Maps}(S,G)$, which comprises all and only the maps $f$ "with finite support" (i.e. such that $X_f:=\{x\in S\mid f(x)\ne 1_G\}$ is finite) is indeed a subgroup of $\operatorname{Maps}(S,G)$. In fact, for every $f,g\in {\rm FS}$ (one-step subgroup test; note that ${\rm FS}\ne\emptyset$, since $e\in {\rm FS}$ because $X_e=\emptyset$ is finite):
\begin{alignat}{1}
X_{f*\tilde g} &= \{x\in S\mid (f*\tilde g)(x)\ne 1_G\} \\
&= \{x\in S\mid f(x)\tilde g(x)\ne 1_G\} \\
&= \{x\in S\mid f(x)g(x)^{-1}\ne 1_G\} \\
&= \{x\in S\mid f(x)\ne g(x)\} \\
&\subseteq X_f\cup X_g
\end{alignat}
which is finite because $X_f$ and $X_g$ are finite. Therefore $f*\tilde g\in {\rm FS}$ for every $f,g\in {\rm FS}$, and hence ${\rm FS}\le\operatorname{Maps}(S,G)$.
Your case is just the special one $G=(\Bbb Z,+)$ and "$*=+$", because then ${\rm FS}$ boils down to your $F\langle S\rangle$. Finally, ${\rm FS}$ is Abelian if $G$ is such, as in your case.