# Relation between different orthogonal groups

I have been reading into orthogonal groups and have some questions.

Recall the orthogonal group of matrices $$O(n)=\{A\in \mathbb{R}^{n,n} : \det{A}\ne 0, A^T=A^{-1}\}$$. This satisfies $$\langle Ax,Ay\rangle=x^TA^TAy=x^Ty=\langle x,y\rangle$$ for all $$x,y\in \mathbb{R}^n$$, where the inner product given is the dot product.

Suppose we have another inner product on $$\mathbb{R}^n$$, say $$[-,-]$$. One can similarly define the orthogonal group of this inner product, $$O([-,-])=\{A\in \mathbb{R}^{n,n}: \det{A}\ne 0,\quad [Ax,Ay]=[x,y]\quad\forall x,y\in \mathbb{R}^n\}$$.

1) From the calculation in paragraph 1, we have $$O(n)\subset O(<-,->)$$ i.e. the orthogonal matrices are a subset of the matrices invariant with respect to the dot product). Can we say that these two definitions are equivalent? From that same calculation in paragraph 1, I only see the requirement that $$A^TA=I_n$$ (i.e. $$A^T$$ is a left inverse). Is this the best we can conclude? (I am pretty sure not!)

2) More generally, can we relate the orthogonal matrices to the arbitrary inner product $$[-,-]$$? Are there any general conditions for which $$O(n)\subset O([-,-])?$$ (or even $$O(n)=O([-,-])?$$

Thoughts: It is well known that any inner product on $$\mathbb{R}^n$$ is of the form $$[x,y]=x^TBy$$, where $$B$$ is a symmetric positive definite matrix. Therefore, if $$A\in O(n)$$, then $$[Ax,Ay]=x^TA^TBAy$$, so I guess the requirement that $$A\in O([-,-])$$ is that $$A^TBA=B$$. That is, $$A^{-1}BA=B$$. From here I have no idea how to proceed further - this still does not feel very intuitive to me!

Edit: it seems $$A^TBA=B$$ characterises the matrices in $$O([-,-])$$ according to Orthogonal matrices only defined for standard inner product?.

Thanks!

• I think you can use the Gram-Schmidt process to construct a basis which is orthonormal with respect to your alternative form $[-,-]$. So under this change-of-basis, $O([-,-])$ is isomorphic to the standard group $O(n)$. – Nick Oct 4 '19 at 0:16
• For 1), note that $A^TA=I_n$ iff $AA^T=I_n$ so the two notions are equivalent. – Mr Martingale Oct 5 '19 at 14:57

The condition $$A^TBA=B$$ is equivalent to $$(B^{-1/2}A^TB^{1/2})(B^{1/2}AB^{-1/2})=I$$. Therefore $$O([-,-])=\left\{B^{-1/2}QB^{1/2}:\ Q\in O(n)\right\}.$$ Now suppose $$A\in O(n)\cap O([-,-])$$. Then $$AB=A(A^TBA)=(AA^T)BA=BA$$. It follows that $$O(n)\subseteq O([-,-])$$ if and only if $$B$$ commutes with all real orthogonal matrices, but this occurs only when $$B$$ is a scalar matrix and in that case, $$O(n)=O([-,-])$$.