What is the meaning of a set being 'stable'? Is this the same as a set being closed under an operation?
To provide some context, the reference I saw is to a set $M$ being 'stable' where there is a law of composition $\circ:M \times M \rightarrow M$. I haven't seen this usage before, is it very common?
No I don't think the usage is common. Michael Artin, for example defines a law of composition on a set $S$ to be "any rule for combining pairs $a, b$ of elements of $S$ to get another element, say $p$ of $S$." (E.g., $p = a\circ b$, $p = a\times b$, $p = a+b$).
More formally, he defines a law of composition as a function of two variables, or a map $\circ : S\times S \to S$, where $S\times S$ denotes the product set, whose elements are pairs of $a, b$ of elements of $S$. I do not see that he defines $S$ as being stable, but given this definition, every pair of elements in $S$ (i.e., a pair $(s_1, s_2) \in S\times S$) maps to (and only to) $S$, and hence for $s_1, s_2 \in S, s_3 = s_1\circ s_2 \in S$.
So to answer your first question:
In this sense, yes, I think one can say that a set $S$ is stable under the binary operation (rule of composition) $*$ if for any $a, b \in S$, $a*b \in S$. I.e., stable, in this sense, meaning $S$ is closed under $*$.
Just curious: What text are you using, or referencing?
Disclaimer: I have answered this based on the tag "abstract algebra", and the (not altogether clear) context in which the term "stable" is used. There are other contexts where one uses "stable groups" in finite group theory, but I suspect that is at a more specialized point than where you encountered "law of composition" (aka, a well-defined binary operation on a set closed under that operation), which is usually early on in abstract algebra, when first defining, say, binary structures like monoids, or groups, etc. There are different contexts altogether (outside of abstract algebra), where the term "stable set" comes up.