maximum value of scalar triple product. 
If $\vec{a}\;,\vec{b}$ are $2$ unit vectors and $\vec{c}$ is a vector such that
$\vec{c} = \vec{a}\times \vec{c}+\vec{b}.$ Then $\max$ of $\vec{a}\cdot (\vec{b}\times \vec{c})$ is

Try: Given $\vec{c}=\vec{a}\times \vec{c}+\vec{b}=-(\vec{c}\times \vec{a})+\vec{b}.$
So $\vec{c}\times \vec{a}=\vec{b}-\vec{c}.$
Now taking dot product with $\vec{b}.$
$\vec{b}\cdot \left(\vec{c}\times \vec{a}\right)=\vec{b}\cdot \vec{b}-\vec{b}\cdot \vec{c}.$
So $[\vec{a}\;\vec{b}\;\vec{c}]=1-|\vec{c}|\cos \theta.$
Could some help me how to solve it, Thanks
 A: $c×a$ is perpendicular to $c$, and $a$ is in the plane perpendicular to $c×a$, so let $$c=(x,0,0),\\c×a=(0,y,0),
\\a=(z, 0,w)$$ 
Use information about the vectors to form equations between $x, y, z$ and $w$.
A: \begin{align}
\vec{c} \times \hat{a}&=\hat{b}-\vec{c}\\
\Rightarrow(\vec{c} \times \hat{a}) \times \vec{c}&=(\hat{b} \times \vec{c})\ - (\vec{c} \times \vec{c})\\
&=\hat{b} \times \vec{c}
\tag{1}
\end{align}
\begin{align}
[\hat{a}.(\hat{b} \times \vec{c})]_{max} 
&=[\hat{a}.(c \ \sin\phi_{(\hat{b},\ \vec{c}\ )}) \ \hat{a}]\\
&=c \sin\phi_{(\hat{b},\ \vec{c}\ )}\\
&=\lvert   \hat{b} \times \vec{c}    \rvert\\
\tag{by 1}
&=\lvert   (\vec{c} \times \hat{a}) \times \vec{c}    \rvert\\
&=\lvert   \vec{c} \times (\vec{c} \times \hat{a})    \rvert\\
&=\lvert   \vec{c} \ (\vec{c}.\hat{a})-\hat{a} \ (\vec{c}.\vec{c})    \rvert\\
&=c^2 \  \lvert   \hat{c} \ (\hat{c}.\hat{a})-\hat{a}    \rvert\\
&=1^2+1^2-2(1)(-1) \cos\theta_{(\hat{c}, \hat{a})}    \\
&=  2c^2 \left[    1+\cos\theta   \right]     \\
&=  2c^2 \left[    1+\cos 2 \left(  \dfrac{\theta}{2}    \right)   \right]    \\
&=  2c^2 \left[    2 \cos^2  \left(   \dfrac{\theta}{2}  \right)   \right]    \\
&=  \left[   2c \cos \left(  \dfrac{\theta}{2}    \right)   \right] ^2    \\
\end{align}
It is somewhat simplified. Hope this is what you are looking for.
A: We have 3 vectors so the problem could be solved in a proper subspace with dimension 3 which contains all the 3 vectors $a,b,c$ . First we solve the problem when $a$ and $b$ are linearly independent i.e. $$a\ne \pm b$$ In this case, we can have the following linear independent basis for solution space $\Bbb R^3$:$$a,b,a\times b$$since $a\times b$ is orthogonal on both $a,b$. Therefore any vector (including $c$) can be expressed as a linear combination of the basis, specially:$$c=k_1a+k_2b+k_3a\times b$$for some $k_1,k_2,k_3\in\Bbb R$. By substituting in the definition ($c=a\times c+b$) we have $$k_1a+k_2b+k_3a\times b=k_2a\times b+k_3a\times(a\times b)+b\\k_1a+k_2b+k_3a\times b=k_2a\times b+k_3(a\cdot b)a-k_3b+b$$which yields to the following system of equations $$\begin{cases}k_1=k_3(a\cdot b)\\k_2=1-k_3\\k_2=k_3\end{cases}$$that has the following answer $$k_1={1\over 2}a\cdot b\\k_2=k_3={1\over 2}$$b substitution we obtain $$c={1\over 2}(a\cdot b)a+{1\over 2}b+{1\over 2}a\times b$$so we have$$a\cdot (b\times c){=a\cdot \left(b\times[{1\over 2}(a\cdot b)a+{1\over 2}b+{1\over 2}a\times b]\right)\\=a\cdot\left({1\over 2}(a\cdot b)b\times a+{1\over 2}a-{1\over 2}b(a\cdot b)\right)\\={1\over 2}\left(1-(a\cdot b)^2\right)}$$This expression is maximum when $a$ and $b$ are orthonormal (which fulfills the assumption of linear independence of $a,b$) and the maximum is $1\over 2$.
A: Here's another proof.
For simplicity and without loss of generality, we can take $\hat a=\hat k$,
then the triple scalar product becomes
$$
I=\hat a\cdot (\hat b\times\vec c)=\hat k\cdot(\hat b\times\vec c)=b_xc_y-b_yc_x
$$
It is given in the problem that $\vec c=\hat a\times \vec c+\hat b$.
We can easily find $c_x=-c_y+b_x$, $c_y=c_x+b_y$, and $c_z=b_z$.
Solving this, we find
$$
c_x=\frac{b_x-b_y}{2},\\
c_y=\frac{b_x+b_y}{2}
$$
Replacing this in the triple scalar product we find
$$
I=\frac{b_x^2+b_y^2}{2}=\frac{1-b_z^2}{2},
$$
since $|\hat b|=1$. Hence, $I$ is maximum if $b_z=0$ and $I_{max}=\frac{1}{2}$.
Confirming the result by Mostafa Ayaz above.
