What does it mean for a subset $A$ of the universe $M$ of an $L$-structure $N$ to be closed under $t^N$ for all $L$-terms $t = t(x_1, ..., x_n)$? Initially I thought it meant closed in the sense of functions i.e if we apply an $L$-term $t$ to some elements of $A$, then the result stays in $A$. But, well, I am not so sure anymore since $L$-terms can be constants, so it doesn't make sense to 'apply' a constant.
 A: 
Initially I thought it meant closed in the sense of functions i.e if we apply an $\mathcal L$-term $t$ to some elements of $A$, then the result stays in $A$.

Quite correct...
The domain $M$ of a structure $\mathfrak A$ is a collection of "objects" (e.g. numbers) and we interpret the language in such a way that formulas are statements about the objects and their properties.
Terms are "names": the individual constant $0$ in the language of arithmetic is the name for the number zero.
To say that a subset $A$ of $M$is "closed under" (the interpretation of term) $t$, for every $t$, means that:

according to the given interpretation, every term $t$ denotes an element of $A$, when its free variables are interpreted in $A$.

This is obvious for individual constants: either $t^{\mathfrak A} \in A$ or not.
But terms are built also from function symbols, like e.g. $+(x,y)$ in the language of arithmetic.
To say that $A$ is closed with respect to it means that every time we assign to variables $x,y$ values in $A$, the interpretation will assign to term $+(x,y)$ a value in $A$.
Some examples with $\mathfrak A = (\mathbb N, 0, s, +, \times)$.
Not all subsets $A$ of $\mathbb N$ are closed under $+^{\mathfrak M}$.
Consider e.g $A = \{ 1,2,3 \}$; we have that for $x=1$ and $y=3$, their sum is not in $A$.
The subset of $\mathbb N: \ A = \{ n \mid n \text { is Even } \}$, instead, is closed with respect to $+^{\mathfrak M}$.
Final example: no proper subset of $\mathbb N$ is closed under the successor function $s(x)$.
