Which manifolds can I glue without worry? Let $X^n$ be a compact $n$-manifold with boundary.  I am interested in when $X$ has the following property: For all compact $n$-manifolds $Y^n$ with $\partial Y \cong \partial X$ and any homeomorphisms $f_1,f_2 : \partial Y \to \partial X$ the resulting closed manifolds $X \cup_{f_1} Y$ and $X \cup_{f_2} Y$ are homeomorphic.  I am also (in fact more) interested in the smooth case where all maps/manifolds are smooth and all homeomorphisms are diffeomorphisms.
For example $B^n$ has this property (by Alexander's trick - although this fails in the smooth case).  I believe that $\natural^k S^1 \times B^3$ also has this property (Laudenbach, François, and Poénaru, Valentin. "A note on 4-dimensional handle bodies.").  Certainly $S^1 \times D^2$ does not since Lens spaces are genus $1$ and there are lots of them.
Is there a nice classification of such manifolds $X$ - perhaps in dimensions $3$ and/or $4$?  Does anyone know any other examples or have references?
 A: Dimension 4 is almost certainly unreasonable unless you make eg a simple connectivity assumption. In dimension two it is true iff $Y$ is orientable and there's only one boundary component or $Y$ is non-orientable.
Dimension 3 is interesting. Suppose $Y$ is oriented and has connected boundary. Of course, if the boundary is a sphere, this is asking whether or not $Y$ has an orientation-reversing homeomorphism; this is true for some spaces and false for others. 
It is false for any higher genus boundary. Adding 2-cells along the boundary you can reduce to the case when the boundary is a torus. I claim that any 3-manifold with torus boundary has infinitely many ways to fill it with a handle $S^1 \times D^2$. Note that, equivalently, after choosing a filling, I'm saying that any knot in a 3-manifold has infinitely many non-homeomorphic surgeries. This can actually be checked by homology alone; infinitely many different homology groups can arise. One could try and make the restriction along the lines of "well, suppose we fix the homology of the surgery...", but I imagine the answer is the same then too.
