How to generate orthogonal matrices? An orthogonal matrix has $\frac{n(n-1)}{2}$ degrees of freedom.
Does there exist a function that maps $\frac{n(n-1)}{2}$ variables to the $n^2$ parameter orthogonal matrix?
And can such a function be differentiable?
The actual objective behind this problem is to feed some input through a random orthogonal matrix and obtain some result which is then fed through a loss function and the gradients are used to optimize the orthogonal matrix. Currently the approach  I'm using is imposing a $L_1(W W^T - I)$ penalty to approximate this - but note how this doesn't prevent $W$ from being non-orthogonal just nudges it in that direction.
 A: An $n \times n$ orthogonal matrix is a matrix whose rows (or columns, if you prefer) form an orthonormal basis of $\mathbb R^n$.  You might start with $n$ random vectors and use Gram-Schmidt to orthonormalize them.  If you run into numerical difficulty (because one of your vectors is too close to the linear span of the previous ones), choose a new random vector to replace it.
A: We consider the Cayley transform: $f:A\in SK_n\rightarrow (I_n-A)(I_n+A)^{-1}\in \Omega_n$ where $SK_n$ is the set of $(n\times n)$ skew-symmetric matrices and $\Omega_n=\{U\in O^+(n)\;;\;-1\notin spectrum(U)\}$. 
$f$ is an algebraic diffeomorphism. It is not onto on $O^+(n)$ but it does not matter if we randomly work on $O^+(n)$. Indeed, a randomly chosen $A\in O^+(n)$ has with probability $1$:
no real eigenvalues when $n$ is even; exactly $1$ real eigenvalue (equal to $1$) when $n$ is odd.
Indeed, consider the function $A\in SK_n\rightarrow e^A\in O^+(n)$ and the fact that a random $A$ is invertible when $n$ is even and has exactly $1$ zero eigenvalue when $n$ is odd.
Then, to make a random choice on $O(n)$, you can proceed as follows:
Randomly choose $A\in SK_n$, calculate $f(A)=[C_1,\cdots,C_n]$; select at random $\pm 1$. If $+$, then output $f(A)$, else output $[-C_1,\cdots,C_n]$.
Of course, the probability law on $O(n)$ will depend on the probability law that you choose on $SK_n$.
A: I've found a way to construct all orthogonal matrices from $\frac{n(n-1)} 2$ variables, and hence all matrices by SVD, as below,

*

*first choose a unit vector from spherical coordinates, with $(n-1)$ variables,
$$v_n=[\cos(x_1), \sin(x_1)\cos(x_2), \sin(x_1)\sin(x_2)\cos(x_3), ..., \sin(x_1)..\sin(x_{n-1})]$$

*construct n-1 orthogonal vectors from $v_n$, based on Istvan Mayer's Simple theorems, proofs, and derivations in quantum chemistry Appendix XI:
first column is $v_1$(sorry I'm not allowed to embed a pic here.)
as a matrix $H_n[n,n-1]$;

*construct another orthogonal matrix $A_{n-1}[n-1, n-1]$, same way as above steps;

*final result recursively:
$$A_n[n, n]=[v_n, H_n*[v_{n-1}, H_{n-1}*[..., H_3*A_2]..]$$
this maps $\frac{n(n-1)} 2$ variables to all orthogonal matrices.
By SVD, we may build a weight matrix with spectral norm strictly controlled, this should be better than regulation/normalization methods, I'm trying this..
