What does it mean for a set $S$ which contains $a$ and $b$ to have an operation $*$ that assigns an element $a*b$? What does it mean for a set $S$ which contains $a$ and $b$ to have an operation $*$ that assigns an element $a*b$?
I think it means for example, if $S = \{a,b,c,\ldots\}$ and the set has the operation $*$ that you can use the operation with the objects in whatever way. So like we can say $a * b$ but we can also say $c * a$.
 A: Like SummerAtlas and Sassatelli commented, a binary operation $*$ on a set $S$ is just a function from $S \times S \to S$
See how $+$ is defined on the set of natural numbers, $\mathbb{N}$
Moreover for any given set $S$ and an operation $*$ defined on $S$, $a*b$ and $b*a$ need not necessarily be equal for $a, b \in S$. For example take the subtraction operation on the set of Integers, $\mathbb{Z}$. See that $(1-2) \neq (2-1) $
A: You've basically got it.
What you've described is called a magma. That's a fancy math word that means "a set of things, plus an operation that eats two of those things and spits out another of those things." That is, the operation exhibits closure: The operation never spits out a thing outside the set entirely.
Other than that, there are no other requirements. As such, magmas are very undemanding entities. Usually, we expect a bit more out of our operation. For instance, if our things are integers and our operation is addition, then we expect additional properties like commutativity ($a+b = b+a$ for any two integers $a$ and $b$), associativity ($(a+b)+c = a+(b+c)$ for any three integers $a$, $b$, and $c$), and the existence of a zero. Some of these seem obvious, but we can define interesting cases where they're not valid. At any rate, these properties make the additive group on the integers special enough to be called a commutative group (often called an Abelian group, after the mathematician Niels Abel).
A major point of defining these entities with various levels of requirements is to see what interesting properties depend on what requirements. For example, provided we define subtraction appropriately, we can ask what properties we rely on to show that subtracting $-a$ is the same as adding $a$. And so on.
