Smallest Transitive Model Containing Certain Sets Let $V$ denote the underlying universe. Let $M \in V$ be a transitive set or class such that $M \models \text{ZF}$. Let $x \in V$ be some set. 
One often sees the notation for $M[x]$ and $M(x)$. If $x$ is a generic filter on a poset, the forcing construction using names gives a construction of $M[x]$. When $x$ is just an arbitrary set, what should $M[x]$ and $M(x)$ mean. 
Taking some of the properties from $L[x]$ and $L(x)$, it seems that one should expect $M[x]$ to be the smallest transitive model of $\text{ZF}$ in $V$ such that $M \subseteq M[x]$ and $x \cap M[x] \in M[x]$. Also $M(x)$ should be the smallest transitive model of $\text{ZF}$ such that $M \subseteq M(x)$ and $x \in M(x)$. 
Is this intended meaning of these $M[x]$ and $M(x)$? Moreover, can $M[x]$ and $M(x)$ be explicitly constructed in a hierarchy like $L[x]$ and $L(x)$? 
 A: Let $\Omega$ denote $\mathsf{ORD}^M$, the height of $M$. This may be $\mathsf{ORD}$ if $M$ is a proper class. If $\Omega=\mathsf{ORD}$, let $V_\Omega$ simply mean $V$. For $\alpha<\Omega$, let $M_\alpha$ denote $M\cap V_\alpha=V_\alpha^M$. Finally, for $t$ a set, let $\mathrm{tc}(t)$ denote the transitive closure of $t$, that is, the smallest transitive set that contains $t$ (as a subset).
Given $x$ whose transitive closure belongs to $V_\Omega$, by $M(x)$ we denote
$$ \bigcup_{\alpha<\Omega}L\bigl(M_\alpha\cup\mathrm{tc}(\{x\})\bigr). $$
As you anticipate, this is indeed the smallest transitive model of $\mathsf{ZF}$ that contains $M$ and has $x$ as an element.
In this sense, our notation for forcing is incorrect, and if $M[x]$ is a forcing extension, instead we should write $M(x)$ -- though, of course, it is too late to hope this will happen. This is probably a remnant of the time when the distinction in the case of $L(x)$ vs $L[x]$ was not followed as carefully as it is now, and one can easily find papers discussing $L[\mathbb R]$ when what is meant is $L(\mathbb R)$.
The classes $M[x]$ and $M(x)$ are sometimes used in contexts where $M$ is $\mathsf{OD}$ or subclass of $\mathsf{OD}$ that is (internally) definable. In those instances, the internal definability of the class $M$ allows us to define $M[x]$ and $M(x)$ by straightforward generalizations of the $L[x]$ and $L(x)$ classes. (See for instance Chapter 13 of Jech's Set theory book.)
I've only seen the notation $M[x]$ used in a setting where the above is not the case, $M[x]$ is not a forcing extension, and $x\notin M[x]$. Namely, when $M$ itself is internally defined, just as $L$, through a hierarchy each level $N'$ of which extends the previous one $N$ by adding certain constants or relations definable over $N$. This is the case when $M$ is a fine structural model over some predicate. For instance, models of the form $L(\mathbb R)[\vec E]$ where $\vec E$ is a coherent sequence of extenders. In these cases, there is no ambiguity with the notation, as we just apply the same distinction as with $L$, and the models are anyway defined through a transfinite recipe that explicitly indicates how each level is formed. But if $M$ is not carefully presented through such a hierarchy, I do not see how one would make sense of the square bracket notation unambiguously. (Note in particular that what Jech calls $M[x]$ in exercise 13.34 is really $M(x)$.)
