Connect Sum is the same as Boundary Connect Sum with punctured manifold I have a fact which I haven't been able to prove. I briefly touched upon it in The first Kirby move and $\mathbb{C}P^2$.
Let $M$ be a manifold-with-boundary, and let $N$ be a closed manifold, both of same dimension.
We have $M\#N$, the connect sum of both.
We have also $M\natural (N-B^4)$, which is done by removing an open 4-ball in $N$, and instead of puncturing the interior of $M$ as in the connected sum we attach $N-B^4$ to $M$ along an embedded $D^3$ of the boundaries of $M$ and $N-B^4$ (which are $\partial M$ and $S^3$).
Claim: The manifolds $M\# N$ and $M\natural (N-B^4)$ are diffeomorphic.
I drew some pictures and it's pretty believable, but I'd like to see some rigorous proof of this.
 A: I have the following pictures: 
Let the manifolds have dimension $4$. Then we can do connect sum of
$M$ and $N$ by cutting out a 4-ball from both,
taking a cylinder $S^3\times I$, then gluing to
$M$ at $S^3\times\{0\}$ and to $N$ at $S^3\times\{1\}$. This is Fig.1.
Now we restrict ourselves to $M$. We find a $D^3\times I$ in $M$ (a band)
such that $D^3\times\{0\}$ lies in the boundary of $M$ and
$D^3\times \{1\}$ lies in the $S^3$ left in the interior of $M$
after removing an open 4-ball. In fact, we can think of this open ball as the same
open 4-ball removed to perform a connect sum with $N$.
(This band exists
because we can take the open 4-ball to lie in the interior of a boundary chart of $M$, so we are doing this in $\mathbb{R}^4_+$).
We now remove this band $D^3\times I$, or more specifically,
$\mathring{D^3}\times I$, leaving $S^2\times I$ behind. This is done in Fig.2.
Claim: Removing this open 4-ball and then the band gives
a manifold diffeomorphic to $M$.
Again, this is easy to see by performing these removals within a boundary coordinate neighborhood of $M$.
Using the claim, we can perform boundary connected sum of $M$
and $N-B^4$ by going from Fig.2 to Fig.3.
In Fig.3 the red
arcs in both $M$ and $N-B^4$ are the $D^3$'s that were
chosen to perform the boundary connected sum.
It is represented as taking a $D^3\times I$ and gluing
$D^3\times\{0\}$ to $M$ along the left $D^3$, and
gluing $D^3\times\{1\}$ to $N$ along the right $D^3$.
The boundary in Fig.3 is composed of the boundary of $M$,
$\partial D^3\times I= S^2\times I$ (the red lines going across),
the boundary of $N-B^4$, but with the two chosen red arcs (the $D^3$'s) removed.
We note this boundary connected sum operation above changed the boundary of $M\sqcup (N-B^4)$ by connect summming the boundary of $M$ with that of
$N-B^4$ which is $S^3$. In particular, the boundary is diffeomorphically
the same as $\partial M$ after the operation.
We can now isotope this resulting manifold by pulling the part of the boundary
lying in the copy of $N-B^4$ towards the copy of $M$, resulting in
Fig.4, at least for the initial stage.
If we keep isotopying the manifold like that, we can make sure that its
boundary is pulled completely into $M$, and then finally back up towards the original boundary of $M$,
following the trace of the band $S^2\times I$ we removed in Fig.2.
This shows that the boundary connected sum $M\natural (N-B^4)$ (Fig.3) is diffeomorphic (through isotopies!) to $M\# N$ (Fig.1), as claimed.
A: Consider handle decompositions, $M = H^0 \cup \dots \cup H^k \cup \dots$ and $N = H'^0 \cup \dots H'^k \cup \dots \cup H'^n$, where $M$ has no $n$-handle because it has boundary, $N$ has one $n$-handle because it's closed, and both have only one $0$-handle. Take their connected sum in $H^0, H'^n$ away from the other handles. Note that $H^0 - D^n$ and $H'n - D^n$ are both cylinders $S^{n-1} \times I $ which we have attached by another such cylinder. If you interpret handle attachments as products of elementary cobordisms, it is clear that $M \# N = H'^0 \cup \dots H'^k \cup \dots \cup H^k \cup \dots $.
Now consider $N - D^n = H'^0 \cup \dots H'^k \cup \dots$. Take the boundary connected sum with $M$ on $\partial H^0, \partial H'^0$ away from the other handles. Convince yourself that a boundary connected sum is the same as attaching a $1$-handle, whose attaching sphere intersects the belt sphere of $H^0$ transversally once. Isotope all of the other handles from $M$ over this $1$-handle, then cancel this $1$-handle with $H^0$. Clearly $M \#_\partial N = H'^0 \cup \dots H'^k \cup \dots \cup H^k \cup \dots.$
