$\vec{u}=\vec{w_{1}}+\vec{w_{2}}, w_1\in W$ and $w_2\in W^\perp$ Let $S[(x_1,y_1,z_1)(x_2,y_2,z_2)]=3x_1x_2+x_2y_1+x_1y_2+y_1y_2+z_1z_2$ be an inner product, 
$W=${$(x,y,z)\in\mathbb{R}^3|x+z=0$} a subspace and $\vec u=(1,0,0)$
Find $w_1\in W,$ $w_2\in W^\perp$ (orthogonal in respect of $S$) such that $\vec u=\vec w_1+\vec w_2$
Here's what I've done so far:
An orthogonal basis of W in respect of the inner product $S$ is $B=${$(1,0,-1),(-\frac{1}{4},1,\frac{1}{4}$)}
If $w=(x,y,z)\in W^\perp$ then $S[(x,y,z),(1,0,-1)]=0$ and $S[(x,y,z),(-\frac{1}{4},1,\frac{1}{4})]=0$ 
So
$3x+y-z=0$ and $\frac{1}{4}x+\frac{3}{4}y+\frac{1}{4}z=0\Rightarrow x+3y+z=0$
Solving the above system we get that the basis for $W^\perp$ in respect of $S$ is $B'=\left \{(1,-1,0),(0,0,1)  \right \}$
I don't know how to proceed from here.. 
Also, are the $\vec w_1$, $\vec w_2:\vec u=\vec w_1+\vec w_2$ unique?
 A: You have found the basis for $W^\perp$ wrongly as those elements do not satisfy $3x+y-z=0$. To compute a solution for 
$$3x+y-z=0$$
$$x+3y+z=0$$
We can simply compute the cross produdct:
$$\begin{bmatrix}  3 \\ 1 \\ -1\end{bmatrix} \times \begin{bmatrix}  1 \\ 3 \\ 1\end{bmatrix}=\begin{bmatrix}4 \\ -4 \\ 8 \end{bmatrix}=4 \begin{bmatrix} 1 \\ -1 \\ 2\end{bmatrix}$$
Hence  a basis of $W^\perp = \left\{ ( 1 , -1 ,2)\right\}$
Let's project $u$ onto $W^\perp$.
$$S[(x_1,y_1,z_1)(x_2,y_2,z_2)]=3x_1x_2+x_2y_1+x_1y_2+y_1y_2+z_1z_2$$
$$S[(1,-1,2),(1,-1,2)]=3(1)(1)+(1)(-1)+(1)(-1)+(-1)(-1)+(2)(2)=6$$
$$S[(1,0,0),(1,-1,2)]=3(1)(1)+(1)(0)+(1)(-1)+(0)(-1)+(0)(2)=2$$
$$w_2=\frac{2}{6}(1,-1,2)=(\frac13,-\frac13,\frac23)$$
$$w_1=w-w_2=(1,0,0)-(\frac13,-\frac13,\frac23)=(\frac23,\frac13,-\frac23)$$
Yes, the decomposition is unique.
A: With
$$
\begin{array}{rcl}
    \vec{u} & = & (1,0,0)\\
\vec{w}_{1} & = & (w_{11},w_{12},w_{13})\\
\vec{w}_{2} & = & (w_{21},w_{22},w_{23})
\end{array}
$$
The inner product is weighted by
$$
M=\left[\begin{array}{ccc}
3 & 1 & 0\\
1 & 1 & 0\\
0 & 0 & 1
\end{array}\right]
$$
The conditions are
$$
\left\{ \begin{array}{rcl}
\vec u & = & \vec{w}_1+\vec{w}_2\\
\left\langle \vec{u},\vec{w}_{1}\right\rangle _{M} & = & \left\Vert \vec{w}_{1}\right\Vert _{M}^{2}\\
\left\langle \vec{u},\vec{w}_{2}\right\rangle _{M} & = & \left\Vert \vec{w}_{2}\right\Vert _{M}^{2}\\
w_{11}+w_{13} & = & 0
\end{array}\right. 
$$
Solving for $\vec{w}_1, \vec{w}_2$ we obtain the solutions
$$
\begin{array}{lccccc}
\vec{w}_1 & = &  \frac{1}{7} \left(4+\sqrt{2}\right) & \frac{1}{7} \left(2-3 \sqrt{2}\right) & \frac{1}{7} \left(-4-\sqrt{2}\right) \\
\vec{w}_2 & = &  \frac{1}{7} \left(3-\sqrt{2}\right) & \frac{1}{7} \left(-2+3 \sqrt{2}\right) & \frac{1}{7} \left(4+\sqrt{2}\right) 
\end{array}
$$
and
$$
\begin{array}{lccccc}
\vec{w}_1 & = & \frac{1}{7} \left(4-\sqrt{2}\right) & \frac{2}{7}+\frac{3 \sqrt{2}}{7} & \frac{1}{7} \left(-4+\sqrt{2}\right) \\
\vec{w}_2 & = & \frac{1}{7} \left(3+\sqrt{2}\right) & \frac{1}{7} \left(-2-3 \sqrt{2}\right) & \frac{1}{7} \left(4-\sqrt{2}\right)
\end{array}
$$
NOTE
Here $< \cdot, \cdot >_M$ is the inner product  $\vec x_1^{\top}M \vec x_2$ and $||\cdot ||_M$ indicated the norm $\sqrt{\vec x^{\top}M\vec x}$
