Finding circle in 3D space from two points and a tangent from one of the points (Essentially this question, but in 3 dimensions.)
In 3D space, points $a$ and $b$ are known. In addition, a unit vector $\hat{t}$ is known. Assume $b-a$ and $\hat{t}$ are not parallel.
The goal: determine the center point of a circle which passes through both $a$ and $b$ and is tangent to $\hat{t}$ at point $a$.
(The circle will then lie on the plane spanned by $b-a$ and $\hat{t}$.)
My attempt:
Let $c$ be the center of the circle (the goal).
Get a vector in the direction of the normal of the plane $n$:
$$n=(b-a)\times\hat{t}$$
Find three equations that can be used to solve for $[c_x, c_y, c_z]$.
First, using the fact that $(a-c)$ and $\hat{t}$ must be perpendicular, $(a-c)\cdot\hat{t}=0$, or
$t\cdot c = t \cdot a \tag{1}$
Second, $(b-c)$ must be perpendicular to $n$:
$n \cdot c = n \cdot b \tag{2}$
Third, $c$ must lie on the perpendicular bisecting plane of $a$ and $b$:
$(b-a)\cdot c = (\frac{b-a}{2})\cdot (b-a) \tag{3}$
This gives the equation
$$
\begin{bmatrix}
t_x & t_y & t_z \\
n_x & n_y & n_z \\
b_x-a_x & b_y-a_y & b_z-a_z
\end{bmatrix}
\begin{bmatrix}
c_x \\
c_y \\
c_z
\end{bmatrix}
=
\begin{bmatrix}
t \cdot a \\
n \cdot b \\
(\frac{b-a}{2})\cdot (b-a)
\end{bmatrix}
$$
However, solving that (by plugging in $a$, $b$, and $\hat{t}$ and doing it on a computer) is giving me wildly off results. $c$ is in the correct plane, but that's about it.
 A: With $p = (x,y,z)$ the circle's center is defined by the intersection of the following three planes
$$
\cases{
\Pi_1\to (p-a)\cdot \vec t = 0\\
\Pi_2\to (p-a)\cdot\left((b-a)\times \vec t\right)=0\\
\Pi_3\to (p-\frac 12(a+b))\cdot(b-a) = 0
}
$$
giving $c$. Now, the circle can be parametrized as
$$
p = c + r\left(\vec e_1\cos\theta+\vec e_2\sin\theta\right)
$$
with
$$
\cases{\vec e_1 = \vec t\\
\vec u = (b-a)-\left((b-a)\cdot \vec t\right)\vec t\\
\vec e_2 = \frac{\vec u}{|\vec u|}\\
r = |a-c|
}
$$
Included a plot when
$$
\cases{
a = (1,1,1)\\
b = (2,3,-2)\\
\vec t = (\frac{1}{\sqrt{2}},0,\frac{1}{\sqrt{2}})
}
$$

A: First $\vec{a}$ and $\vec{b}$ must lie in the plane you mentioned, but such that both are on the same side of $\hat{t}$. Set
$$
\vec{m} = \vec{b} - \vec{a}
$$
Second, form the vector
$$
\hat{n} = \frac{\hat{t} \times (\hat{t} \times \hat{m})}{||\hat{t} \times(\hat{t} \times \hat{m})||} = \hat{t} \times \frac{\hat{t} \times \hat{m}}{||\hat{t} \times \hat{m}||}
$$
where $\hat{m}=\vec{m}/||\vec{m}||$. The vector $\hat{n}$ points to the center $\vec{c}$ and is normalized. Now dot it with $\hat{m}$ you get
$$
\hat{n} \cdot \hat{m} = \cos \alpha
$$
where $0 \leq \alpha \leq \frac{\pi}{2}$ is the angle between the $\hat{m}$ and $\hat{n}$, and hence from $\hat{b}$ to the diameter.
Now since $\vec{m}$ is a chord in the circle, it forms a right triangle with the diameter, which forms the hypotenuse. It follows that
$$
2R = \frac{||\vec{m}||}{\cos \alpha}
$$
and finally
$$
\vec{c} = \vec{a} + R\ \hat{n} = \vec{a} + \frac{||\vec{m}||}{2\cos \alpha}\ \hat{n}
$$
A: Equation (3) should be:
$$(b-a)\cdot (c-a)=\Big(\frac{b-a}{2}\Big)\cdot(b-a)\tag 3$$
