Singular values of square orthogonal matrix? What are the singular values of an $n \times n$ square orthogonal matrix? 
How do we know that the set of all orthogonal matrices is convex? Is there an example?
 A: SVD of a matrix $A$ is $A = U \Sigma V^T$, where $U$ and $V$ are orthogonal and $\Sigma$ is nonnegative real diagonal.
Now, let $X$ be orthogonal. Note that $X = U \Sigma V^T$, where $U := X$ is orthogonal, $\Sigma := {\rm I}$ is diagonal, and $V := {\rm I}$ is orthogonal. So, singular values are all equal to $1$.
Or, you can use the definition by which the singular values of $X$ are the absolute square roots of the eigenvalues of $X^TX$. In case of an orthogonal $X$, eigenvalues of $X^T X = {\rm I}$ are all equal to one, so the singular values of $X$ are all equal to $1$.
As for your second question, I don't think the statement is true. Let
$$X = {\rm I} = \begin{bmatrix} 1 \\ & 1 \end{bmatrix}, \quad Y = \begin{bmatrix} & 1 \\ 1 \end{bmatrix}.$$
If the set of the orthogonal matrix is convex, then $Z := \frac{1}{2}(X+Y)$ is also orthogonal. But,
$$Z = \frac{1}{2}(X+Y) = \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$
is singular, so it cannot be orthogonal. We can even check directly: $Z^T Z = Z \ne {\rm I}$.
You can find a topic on the convex hull of the set of orthogonal matrices here.
