# Proof of the rotation matrix is an extreme point of $\text{conv } SO(n)$

Define the set of rotation matrices:
\begin{aligned} SO(n) := \{X\in \textbf{R}^{n\times n}: X^TX=I, \text{det}(X)=1\} \end{aligned}

I want to prove that if $X\in SO(n)$, $X$ is an extreme point of conv $SO(n)$.

My work:

1. Suppose not. Then $X=(X_a+X_b)/2$ where $X_a,X_b\in$ conv $SO(n)$, $X_a,X_b$ are defined at the bottom.
2. By the definition of $SO(n)$, we have the following

\begin{align*} &\big(\frac{X_1+Y_1}{2}\big)^T\big(\frac{X_1+Y_1}{2}\big)=I \\ \Rightarrow \ \ & \frac{1}{4}X_1^TX_1+\frac{1}{4}X_1^TY_1+\frac{1}{4}Y_1^TX_1+\frac{1}{4}Y_1^TY_1=I \\ \Rightarrow \ \ & \frac{1}{4}(Y_1^TX_1 + X_1^TY_1)=\frac{1}{2}I \\ \Rightarrow \ \ & Y_1^TX_1+X_1^TY_1 = 2I \end{align*}

I have no idea how to come up with the contradiction.

The definition of conv $SO(n)$:

According to @stewbasic suggestion:

If $X_a, X_b\in$ conv $SO(n)$, then $X_a=\sum \theta_i X_i$ and $X_b=\sum \alpha_i Y_i$ where $X_i,Y_i\in SO(n)$. And $\sum \theta_i = 1, \sum \alpha_i=1$ and $\theta_i , \alpha_i \geq 0$. Consider the simplest case, $X_a=X_1$ and $X_b=Y_1$ with $X_1,Y_1\in SO(n)$. And go back to the proof above.

• The definition of the convex hull of $SO(n)$ is the set $\{a_1X_1+\ldots+a_kX_k:X_i\in SO(n),\,\sum_ia_i=1,\,a_i\geq0\}$. The theorem you state gives an alternate description, but I think the definition will be more useful for this question. In particular, you should be able to show that $\|Xv\|\leq\|v\|$ for $X$ in conv $SO(n)$ and $v\in\mathbb R^n$. – stewbasic Jun 30 '16 at 23:14
• @stewbasic I think it is the following $\|X\mu\|^2=(X\mu)^T(X\mu)=\mu^TX^TX\mu=\mu^T\mu=\|\mu\|^2$. – sleeve chen Jun 30 '16 at 23:22
• That shows $\|Xv\|=\|v\|$ for $X\in SO(n)$. What can you say if $X\in\text{conv }SO(n)$? – stewbasic Jun 30 '16 at 23:25
• using your expression $X_a=\sum\theta_iX_i$, try to show $\|X_av\|\leq\|v\|$. – stewbasic Jul 1 '16 at 0:52
• I was suggesting you use both the property of extreme points and the one I mentioned. If $\|X_av\|,\,\|X_bv\|\leq\|v\|$ and $\|\frac12(X_a+X_b)v\|=\|v\|$ for all $v$, then $X_a=X_b$. – stewbasic Jul 1 '16 at 1:22