Define the set of rotation matrices:
\begin{equation} \begin{aligned} SO(n) := \{X\in \textbf{R}^{n\times n}: X^TX=I, \text{det}(X)=1\} \end{aligned} \end{equation}

I want to prove that if $X\in SO(n)$, $X$ is an extreme point of conv $SO(n)$.

My work:

  1. Suppose not. Then $X=(X_a+X_b)/2$ where $X_a,X_b\in $ conv $SO(n)$, $X_a,X_b$ are defined at the bottom.
  2. By the definition of $SO(n)$, we have the following

\begin{align*} &\big(\frac{X_1+Y_1}{2}\big)^T\big(\frac{X_1+Y_1}{2}\big)=I \\ \Rightarrow \ \ & \frac{1}{4}X_1^TX_1+\frac{1}{4}X_1^TY_1+\frac{1}{4}Y_1^TX_1+\frac{1}{4}Y_1^TY_1=I \\ \Rightarrow \ \ & \frac{1}{4}(Y_1^TX_1 + X_1^TY_1)=\frac{1}{2}I \\ \Rightarrow \ \ & Y_1^TX_1+X_1^TY_1 = 2I \end{align*}

I have no idea how to come up with the contradiction.

The definition of conv $SO(n)$:

http://arxiv.org/pdf/1403.4914v1.pdf (p.1315)

enter image description here

According to @stewbasic suggestion:

If $X_a, X_b\in$ conv $SO(n)$, then $X_a=\sum \theta_i X_i$ and $X_b=\sum \alpha_i Y_i $ where $X_i,Y_i\in SO(n)$. And $\sum \theta_i = 1, \sum \alpha_i=1$ and $\theta_i , \alpha_i \geq 0$. Consider the simplest case, $X_a=X_1$ and $X_b=Y_1$ with $X_1,Y_1\in SO(n)$. And go back to the proof above.

  • 2
    $\begingroup$ The definition of the convex hull of $SO(n)$ is the set $\{a_1X_1+\ldots+a_kX_k:X_i\in SO(n),\,\sum_ia_i=1,\,a_i\geq0\}$. The theorem you state gives an alternate description, but I think the definition will be more useful for this question. In particular, you should be able to show that $\|Xv\|\leq\|v\|$ for $X$ in conv $SO(n)$ and $v\in\mathbb R^n$. $\endgroup$ – stewbasic Jun 30 '16 at 23:14
  • $\begingroup$ @stewbasic I think it is the following $\|X\mu\|^2=(X\mu)^T(X\mu)=\mu^TX^TX\mu=\mu^T\mu=\|\mu\|^2$. $\endgroup$ – sleeve chen Jun 30 '16 at 23:22
  • 1
    $\begingroup$ That shows $\|Xv\|=\|v\|$ for $X\in SO(n)$. What can you say if $X\in\text{conv }SO(n)$? $\endgroup$ – stewbasic Jun 30 '16 at 23:25
  • 1
    $\begingroup$ using your expression $X_a=\sum\theta_iX_i$, try to show $\|X_av\|\leq\|v\|$. $\endgroup$ – stewbasic Jul 1 '16 at 0:52
  • 1
    $\begingroup$ I was suggesting you use both the property of extreme points and the one I mentioned. If $\|X_av\|,\,\|X_bv\|\leq\|v\|$ and $\|\frac12(X_a+X_b)v\|=\|v\|$ for all $v$, then $X_a=X_b$. $\endgroup$ – stewbasic Jul 1 '16 at 1:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.