I recently came across a methodology for orthogonalizing variables that are collinear, that uses Cholesky Decomposition, but I am not entirely grasping the intuition of it.

Let' assume we have three variables/factors : $X_1$, $X_2$, and $X_3$. Let $V$ be a full rank matrix, of which the columns are $X_1$, $X_2$ and $X_3$. We wish to orthogonalize the columns of the matrix $V$.

Here is the methodology:

We have that $V'V$ is Hermitian positive definite, then, using Cholesky Decomposition, we can write:

$V'V = LL'$

The lower triangular matrix $L$ with strictly positive diagonal entries is invertible. Then the columns of the matrix:

$U = V(L^{-1})'$ are orthonormal

Can someone please explain why this last result holds?

Thanks a lot!

  • $\begingroup$ In general, if $AA^* = BB^*$ is invertible, then $A^*(B^*)^{-1} = A^{-1}B$ is orthogonal. This can be verified by direct calculation, using the definition of an orthogonal matrix ($Q$ is orthogonal means $QQ^* = I$). $\endgroup$ – Alexey Mar 20 '17 at 17:24

What you described is a very indirect way of computing the QR decomposition (aka the Gram Schmidt process). Let us write the QR decomposition as $V = QR$. Then $V'V = R'Q'QR = R'R = LL'$. Therefore to recover the orthonormal $Q$, we can do $Q = VR^{-1}$, which is the last result. You're better off computing $V = QR$ directly.

  • $\begingroup$ Isn't the Gram-Schmidt process successive? In order words, the order in which the orthogonalization is done matters. The intuition is that you choose a vector to start with, then subtract the projection of that vector from each of the other vectors (hence the importance of the order). I thought that Cholesky decomposition is way to go around this problem, isn't it? $\endgroup$ – Mayou Sep 8 '14 at 18:57
  • $\begingroup$ Also, is there any other method of "orthogonalizing" the columns of V, but not in a successive manner, but rather in a "simultaneous" manner where the order doesn't matter? Thanks! $\endgroup$ – Mayou Sep 8 '14 at 19:12
  • $\begingroup$ To answer your first comment above, yes, the process is successive, and the ordering of the vectors matters. The Cholesky decomposition is completely equivalent to Gram Schmidt in the way you've described it. $\endgroup$ – Victor Liu Sep 8 '14 at 19:16
  • $\begingroup$ To answer your second comment, a set of orthogonal vectors spanning the same subspace as a given set of vectors will always be somewhat arbitrary. There's no (easily computed) "canonical" set of orthogonal vectors derived from the given set (do you care about the output order of the orthogonal vectors?). You might say that you want the output orthogonal vectors to be "as close as possible" to the input set, in some sense, but now you have to solve a pretty hard optimization problem. The QR decomposition gives a very efficient solution to the problem. $\endgroup$ – Victor Liu Sep 8 '14 at 19:18
  • $\begingroup$ How about Mayers' Othogonalization? $\endgroup$ – Mayou Sep 8 '14 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.