Orthogonal vector in a plane

I have two vectors $\boldsymbol{a}$ and $\boldsymbol{b}$. Now I want to find a vector $\boldsymbol{v}$ orthogonal to $\boldsymbol{a}$ that is in the plane spanned by $\boldsymbol{a}$ and $\boldsymbol{b}$. Is it ok if I do $\boldsymbol{v} = \boldsymbol{a} \times (\boldsymbol{a} \times \boldsymbol{b})$?

Yes. $\boldsymbol a\times (\boldsymbol a\times \boldsymbol b)$ is perpendicular to both $\boldsymbol a$ and $\boldsymbol a\times \boldsymbol b$. Being perpendicular to $\boldsymbol a\times \boldsymbol b$ means being on the plane generated by $\boldsymbol a$ and $\boldsymbol b$.

See the other answer for a less expensive computation that outputs such a vector.

• Can't you just do $a \times b$? Why the $a\times(a\times b)$ Mar 21 '18 at 23:44
• Because $a\times b$ does not belong to the plane Mar 21 '18 at 23:48
• @KSplitX In fact, $\boldsymbol a \times \boldsymbol b$ is actually perpendicular to the plane. Mar 22 '18 at 0:46

As @Tal-Botvinnik says, it's perfectly OK in general. There is a failure case, which is when $a$ and $b$ are linearly dependent; then $a \times b = 0$, and you're in trouble.

A computationally slightly cheaper approach is to take $$n = b - \frac{b \cdot a}{a \cdot a} a.$$

This also works in the case where $a$ and $b$ are vectors in any inner-product space, not just 3-dimensions.

(BTW, it also fails on the same case as the cross-product approach; that's because the answer isn't unique in that case.)