Building orthogonal vectors

Question

Suppose I have $N$ vectors $v_1, v_2, v_3, ..., v_N$

I already know that I can build an orthonormal basis with the Gram-Schmidt process, but I need to keep the first three fixed, even if they are not orthogonal, i.e. I would like to build vectors in this way:

• $q_1 = \frac{v_1}{\left\lVert{v_1}\right\rVert_2}$
• $q_2 = \frac{v_2}{\left\lVert{v_2}\right\rVert_2}$
• $q_3 = \frac{v_3}{\left\lVert{v_3}\right\rVert_2}$
• $q_4$ orthogonal to $q_1$, $q_2$, $q_3$
• $q_5$ orthogonal to $q_1$, $q_2$, $q_3$, $q_4$

...

• $q_N$ orthogonal to $q_1$, $q_2$, $q_3$, ..., $q_{N-1}$

I've tried using the Gram-Schmidt process, but the fact that $q_1$, $q_2$, $q_3$ are not orthogonal invalidates the Gram-Schmidt procedure.

Any suggestions how I could proceed?

Answer 1

You can build a linear least squares problem like this

$${\bf q_k} = {\bf {\hat q}_k + v} \,\,s.t.\,\, \min_{\bf v}\sum_{l=1}^{k-1} \| {\bf q_l}^t ({\bf {\hat q_k + v}})\|_2^2$$

Where $\bf \hat q_k$ is a candidate which for example can be randomized as in the case of Gram-Schmidt.

In the case of a ON subspace, the Gram-Schmidt process actually does precisely this. It is just that as we know the previous ones are all orthogonal to one another, the minimum will be found by removing a set of projections as in the case of ON subspace the projections will never risk removing portions of vectors which lie in any dimension where the other basis vectors measure length.

We may do well to add a small regularizing term $+\epsilon\|{\bf v}\|_2^2$ to make sure that we don't end up with $\bf \hat q_k +v$ being the zero vector.