Vector equation of a plane through three known points. I need the equation in scalar product form and without using matrices (Simultaneous instead).
The three points:  $A (2,3,5)$,  $B (-1,0,3)$,  $C (4,4,1)$.
I want to use the method where: the normal to the plane is normal to the three vectors $\overrightarrow{AB}$, $\overrightarrow{BC}$ and $\overrightarrow{CA}$.
So we get the three vectors $\overrightarrow{AB}$, $\overrightarrow{BC}$ and $\overrightarrow{CA}$ out of the points.
For example for the vector $\overrightarrow{AB}$ we do $A-B$ ... so $\begin{pmatrix}2\\3\\5\\ \end{pmatrix}$-$\begin{pmatrix}-1\\0\\3\\ \end{pmatrix}$=$\begin{pmatrix}3\\3\\2\\ \end{pmatrix}$
...so:
$\overrightarrow{AB}$= $\begin{pmatrix}3\\3\\2\\ \end{pmatrix}$
$\overrightarrow{BC}$= $\begin{pmatrix}-5\\-4\\2\\ \end{pmatrix}$
$\overrightarrow{CA}$= $\begin{pmatrix}2\\1\\-4\\ \end{pmatrix}$
Now let's call the normal to the plane "$n$".
Since $n$ is normal to all those 3 vectors so:
$\overrightarrow{AB} \cdot n=0$ and $\overrightarrow{BC} \cdot n=0$ and $\overrightarrow{CA} \cdot n=0$
Now we can get 3 equations with 3 unknowns out and solve them to get the normal.
Let the 3 components of $n$ be $s$, $t$ and $u$ or $\begin{pmatrix}s\\t\\u\\ \end{pmatrix}$.
So, by taking the dot product, the three equations are:
$3s+3t+2u=0$
$-5s-4t+2u+0$
$2s+t-4u=0$
I'm stuck here! I can't solve those 3 equations. I can't get except one equation out of them. ...Where's my mistake? What's worng?
 A: Nothing's wrong. Well,one thing is wrong: the vector $AB$ is $B - A$, not $A - B$, but that won't affect the result. And as Danny notes, the usual solution is to use the cross-product, but I'll follow your approach, since that's what you asked for. 
The three equations are redundant: the third is a consequence of the first two. That'll mean that you don't get a unique solution, which is good, because if $\mathbf n$ is a normal to your plane, so is $c\mathbf n$ for any $c \ne 0$. 
How do you solve an equation like this when there's one free variable? I pick two of the equations, and row reduce: 
$$
3s + 3t + 2u = 0\\
2s + t - 4u = 0
$$
Subtract second from first:
$$
s + 2t + 6u = 0\\
2s + t - 4u = 0
$$
Subtract twice first from second:
$$
s + 2t + 6u = 0\\
0s -3t - 16u = 0
$$
Now you pick a value for $u$ -- say $3$, to make the arithmetic nice, and get 
$$
s + 2t + 6\cdot 3 = 0\\
0s -3t - 16\cdot 3 = 0
$$
so
$$
s + 2t = -18\\
-3t =  48
$$
Hence $t = -16$, so 
$$
s - 2\cdot 16 = -18
$$
$$
s =  32 -18 = 14
$$
So a solution  is $(14, -16, 3)$. 
Why pick a value for $u$ rather than $s$ or $t$? Because after row-reduction, $s$ and $t$ were leading nonzero entries in some equation. 
