Terminology: Semigroups, only their "binary operations" aren't closed. Motivation:
Consider $\mathcal{X}=(X, +)$, where $X=\{-1, 0, 1\}$ and $+$ is standard addition. Then $\mathcal{X}$ is associative (where defined) but not closed.
NB: There is an identity element in $X$ and inverses exist in $X$ all with respect to $+$.
This example is taken from here.
The Question:
This question seems difficult to pose due to certain subtleties so, to make life easier, here's the rough idea first.

What d'you call a "magma" that's associative but not closed?

An attempt at refining the question:

What do you call the mathematical objects $\mathcal{S}=(S, T, \ast)$ for which $S$ is a set and $\ast$ is some function with domain $S\times S$ and codomain some set $T$ with $S\subset T$, such that

*

*for all $s,t,u\in S$ we have $$s\ast (t\ast u)=(s\ast t)\ast u$$ whenever $t\ast u, s\ast t\in S$ (or $T$ if that's necessary to keep the question in spirit) and


*there exist $x, y\in S$ such that $x\ast y\in T\setminus S$?

(Please disregard this attempt if it complicates the idea of the question needlessly.)
Thoughts:
I'm not sure whether naming these things is necessary. I'm interested in them out of curiosity. Whether the question even makes sense, I don't know.

Are they simply subsets of semigroups?

I made sure to say function and not binary operation above, since the latter implies closure by definition.
 A: You might have put together two different questions due to some subtleties. As I see it, you have two main possibilities. Starting from the one you wrote explicitly:


*

*A subset $S$ of a magma $(T,\cdot)$ for which 
$$ (s\cdot t)\cdot u = s\cdot (t\cdot u)  \qquad \forall s,t,u\in S $$


Note that this does not raise any doubts about existance of the products as the equality holds in $T$.
I'm not sure this has a name but a fair bet would be something like:  " $S$ is an associative subset of the magma $T$ " or maybe " $T$ is locally associative over the subset $S$ ".


*A set $S$ with a partial operation $\star: S\times S \rightarrow S$ for which, for all $s,t,u\in S$, 
$$  \text{ if } s\star t\text{ , }t\star u\text{ , }(s\star t)\star u \text{ , }s\star(t\star u)\in S \qquad \text{then } (s\star t)\star u = s\star(t\star u) $$
(Meaning that: if all the products involved exist, then $\star$ is associative)
This seems to be quite similar to the notion of a partial semigroup. 
Using this view you could also see $S$ in definition 1. as a "partial subsemigroup" of the magma $T$ (similarly to what is done for the notion of a subgroup of a semigroup).
A: It is not really an answer to your question, but I could not enter the picture in a comment. The point that if you "duplicate" the zero into $0_A$ and $0_B$, you could define a category which is very close to your structure:
$\hskip 15pt$  
Now you have an associative structure with $1 + (-1) = 0_A$ and $(-1) + 1  = 0_B$.
A: Such a structure is not always a subset of a semigroup.
Take the following structure:
ab = ba = c
bc = cb = a
ca = ac = b
aa = bb = cc = e
ee = f
This is basically the Klein group, except that I do not define ea, eb, ec, ef and that I willfully set ee different from e.
Now, this is clearly associative, because the only problem e cannot interact with the rest due to lack of definition. But when I want to expand the structure to a hopefully closed associative structure I get
$ae = a (bb) = (ab)b = cb =a $, therefore $f= ee = (aa)e= a(ae) = aa=e$ which is a contradiction.
