How to tell if a point is between two others in $\mathbb{R}^3$ 
Find the coordinates $c$ of the point $C$ $st$ $C$ is on the line from
   $A (2, -3, 1)$ to $B (8, 9, -5)$, it is between A and B, and $\vec{AC}
 = 2\vec{CB}$.

So, I found the vector $\vec{AB}$ to be $(6, 12, -6)$ and from the equality, I set up the following:
$
\left(\begin{array}{cc}
c_1 - 2\\
c_2 +3\\
c_3 - 1\\
\end{array}\right)=
\left(\begin{array}{cc}
16 - 2c_1\\
18 - 2c_2\\
-10-2c_3\\
\end{array}\right)
$
to get $c_1 = 6$, $c_2 = 7$, $c_3 = 3$
but I wasn't sure how to tell if $(6, 7, 3)$ was in between A and B and I also had no use for $\vec{AB}$ which seemed strange.
How do I solve this problem? and how do I know if a point is between two others generally in dimensions greater than $\mathbb{R}^2$?
 A: Your solution would have been fine if you’d solved correctly for $c_1,c_2$, and $c_3$: you should have got $c_2=5$ and $c_3=-3$.
Yours is the more straightforward approach, but here’s another way to think about it.
Points on the line through $A$ and $B$ are precisely those that you can reach by going from the origin to $A$ and then some distance, possibly negative, in the direction $\overrightarrow{AB}$. In other words, they correspond to vectors of the form $\vec a+\lambda\vec v$, where $\vec a=\langle 2,-3,1\rangle$ and $\vec v=\langle 6,12,-6\rangle$. You want the point that is two-thirds of the way from $A$ to $B$, so you want $$\vec a+\frac23\vec v=\langle 2,-3,1\rangle+\frac23\langle 6,12,-6\rangle=\langle 6,5,-3\rangle\;.$$ Thus, $C$ should be the point $\langle 6,5,-3\rangle$.
As a check, $|AB|=\sqrt{6^2+12^2+(-6)^2}=\sqrt{216}=6\sqrt6$, $|AC|=\sqrt{4^2+8^2+(-4)^2}=\sqrt{96}=4\sqrt6$, and $CB=\sqrt{2^2+4^2+(-2)^2}=\sqrt{24}=2\sqrt6$, exactly as desired.
A: The points $C$ between $A$ and $B$ can be parametrized by
$$
C=At+B(1-t)\tag{1}
$$
where $0< t< 1$
Therefore, we wish to solve for $t\in(0,1)$ so that
$$
\begin{align}
|C-A|&=2|C-B|\\
|At+B(1-t)-A|&=2|At+B(1-t)-B|\\
|(B-A)(1-t)|&=2|(A-B)t|\\
1-t&=2t\\
&t=\tfrac13\tag{2}
\end{align}
$$
Thus, plugging $(2)$ into $(1)$ yields
$$
\begin{align}
C
&=\tfrac13A+\tfrac23B\\
&=\tfrac13(2, -3, 1)+\tfrac23(8, 9, -5)\\
&=(6,5,-3)\tag{3}
\end{align}
$$

Determining if $C$ is between $A$ and $B$:
In general, $C$ is between $A$ and $B$ when
$$
|B-A|=|B-C|+|C-A|\tag{4}
$$
Equation $(4)$ is the extreme case of the triangle inequality, which says
$$
|B-A|\le|B-C|+|C-A|\tag{5}
$$

Check:
Let's check if $C=(6, 7, 3)$ is between $A=(2, -3, 1)$ and $B=(8, 9, -5)$:
$$
|A-B|=|(-6,-12,6)|=6\sqrt{6}
$$
but
$$
|A-C|+|C-B|=|(-4,-10,-2)|+|(-2,-2,8)|=2\sqrt{30}+6\sqrt{2}
$$
Numerically, $6\sqrt{6}\approx14.70$ and $2\sqrt{30}+6\sqrt{2}\approx19.44$. Thus,
$$
(6, 7, 3)\text{ is not between }(2, -3, 1)\text{ and }(8, 9, -5)\tag{6}
$$
Let's check if $C=(6, 5, -3)$ is between $A=(2, -3, 1)$ and $B=(8, 9, -5)$:
$$
|A-B|=|(-6,-12,6)|=6\sqrt{6}
$$
and
$$
|A-C|+|C-B|=|(-4,-8,4)|+|(-2,-4,2)|=4\sqrt{6}+2\sqrt{6}=6\sqrt{6}
$$
Thus,
$$
(6, 5, -3)\text{ is between }(2, -3, 1)\text{ and }(8, 9, -5)\tag{7}
$$

Extension:
Given this question, one might wonder what is the set of all points $C$ so that 
$$
|C-A|=2|C-B|\tag{8}
$$
Squaring $(8)$ yields
$$
C\cdot C-2A\cdot C+A\cdot A=4C\cdot C-8B\cdot C+4B\cdot B
$$
$$
0=3C\cdot C-2(4B-A)\cdot C+4B\cdot B-A\cdot A
$$
Therefore,
$$
\begin{align}
0
&=C\cdot C-2\left(\tfrac43B-\tfrac13A\right)\cdot C+\tfrac43B\cdot B-\tfrac13A\cdot A\\
&=\left|C-\left(\tfrac43B-\tfrac13A\right)\right|^2+\left(\tfrac43B\cdot B-\tfrac13A\cdot A\right)-\left(\tfrac43B-\tfrac13A\right)\cdot\left(\tfrac43B-\tfrac13A\right)\\
&=\left|C-\left(\tfrac43B-\tfrac13A\right)\right|^2-\tfrac49|A-B|^2\tag{9}
\end{align}
$$
So the set of points that satisfy $(8)$ is the sphere centered at $\tfrac43B-\tfrac13A$ with radius $\frac23|A-B|$.
