If you can find the reflection through the plane perpendicular to the middle of the $AB$ segment, then it automatically sends $A$ to $B$ and $B$ to $A$.
How do we do this reflection? First, suppose that the midpoint of $A$ and $B$ is the origin: $A=-B$. Now, note the mathematical concept of a projection:
Here, $u,v,w,x$ are being multiplied by a projection matrix $P$: one that sends them to their closest point on the line $m$. It turns out that the projection of a vector Now suppose that $m$ is the line from $A$ to the origin.
If we subtract the projection of the line $x$ on to $OA$ from $x$ twice, then we get the reflection of $x$ in the plane through $O$ perpendicular to $OA$. Now, if $a$ is the vector of the point $a$, then the projection of the vector $x$ on to the line $a$ is given by $(x.a)x$ (do you know how to work out the dot product?) If you know properties of the dot product, you can work this out for yourself. So the transformation that reflects $x$ in this plane is given by:
This is a linear transformation, so it has a matrix, if you want to work it out: to work out the matrix, substitute in $(1,0,0)$, $(0,1,0)$ and $(0,0,1)$ for $x$, and the vectors you get out will be the columns of the matrix.
What if $A\neq-B$? Then, as I said in my comment above, there is no matrix that will encode the reflection, as any matrix must send the origin to itself. So we have to use a trick called conjugation. First, find the midpoint $c=\frac12(a+b)$, and then let $a'=a-c$ and $b'=b-c$, and for any point $x$, let $x'=x-c$. So we are 'changing our world view' in some sense: we are taking the point $c$ and mapping it to the origin. Then $a'+b'=0$, so we can encode the reflection in the plane between them as a matrix $P'$.
Then, if $x$ is an arbitrary vector, the reflection of $x'$ in the plane between $a'$ and $c'$ is given by $P'x'=P'(x-c)$. If we now add $c$ to everything, we see that the reflection of $x$ in the plane between $a$ and $b$ is given by $P'(x-c)+c$, which we can rearrange to $P'x+d$, where $d=c-P'c$. So the transformation isn't given by a matrix, but it is given by a matrix multiplication, followed by adding a constant vector.
The process of carrying out an operation (subtracting $c$), then carrying out another operation (the reflection) and then doing the inverse of the first operation (adding $c$ back again) is known as conjugation, and is very important in mathematics. Think of it as a temporary change of 'world view': we pretend that $c$ is the origin so we can do our reflection, and then we map everything back again at the end.