So I actually solved this problem using an iterative solver, but it annoys me because as far as I can tell it should be possible to do it directly.
I have three known 3D "rays" that all start at the origin, and are represented as three unit vectors $a$, $b$, and $c.$
I know that these three rays all touch the surface sphere of known radius $R$ (without intersecting the interior of the sphere). So the task is to find the center position of the sphere $p$.
For this to have a solution the three rays all point more or less in the same direction which I know they do by construction in my case (in fact, in my specific case I happen to know that the $z$ component is $<0$ for $a, b$ and $c$). I also know that the sphere is "in front" of all thee rays (and does not contain the origin.. i.e. $p.p > R^2$)
I thought this would be easy to solve, just set up the distance from the point to each ray and set it to be equal to $R$, then manipulate and solve for $p$, but alas I could not manage to isolate any sensible expression for $p$. I then tried both Maple and Mathematica and was unable to solve it there either (in fact, Mathematica just hangs indefinitely). This leads me to believe I'm not properly stating this problem.
As I mentioned, I was able to solve this iteratively (gauss newton) but it just kind of bugs me so I was hoping maybe someone could have a stab and show me how to do it, maybe I'll pick up some tricks for next time.