What's the interpretation of a unitary matrix? I know that a unitary matrix is a matrix whose inverse equals its conjugate transpose (or that multiplying it by its conjugate transpose yields the identity), but I don't have a deep intuition about it (I just accept the definition). So for example, when I encounter the statement that the left and right singular decompositions $U$ and $V$ in the SVD are unitary, I don't get the significance. I would appreciate if somebody could enlighten me to connect the dots and how to feel when encountering unitary matrices. I have the feeling there is something unwritten that I'm missing.
 A: The unitary matrices are precisely those matrices which preserve the Hermitian inner product. 
$$\langle v,w \rangle = \langle Uv,Uw \rangle$$
Hence it preserves the distance or length of a vector in the Unitary space (the finite-dimensional vector space over the complex number field $\mathbb{C}$ with an inner product) under rotation or reflection.
A: Structurally, unitary matrices are rotations and reflections. Perhaps it's more clear to first picture unitary diagonalization before the singular value decomposition. Suppose we unitarily diagonalize
$$A = UDU^{\dagger}$$
In unitary diagonalization, we first rotate (and possibly reflect) from our standard basis into our new orthonormal basis. This is the action of $U^{\dagger}$. Then we perform stretches by the magnitudes of the eigenvalues in the respective basis directions. This is the action of the diagonal matrix $D$. Finally we rotate back to our original basis, which is the action of $U$ which reverses $U^\dagger$.
The action of a singular value decomposition is virtually identical, except that the "diagonal" matrix $\Sigma$ does not necessarily map the same space to itself, so that the rotations happen in different vector spaces.
A: Unitary matrices are the complex analogues of orthogonal matrices, and both are very common in the theory of Lie groups and Lie algebras. Orthogonal matrices are the matrix representations of real linear maps that preserve distance. Unitary matrices are the complex versions, and they are the matrix representations of linear maps on complex vector spaces that preserve "complex distances".
If you have a complex vector space then instead of using the scaler product like you would in a real vector space, you use the Hermitian product. The Hermitian product of two complex vectors (thought of as $n$-by-$1$ matrices), say $v$ and $w$, is defined to be $\langle v,w\rangle = \overline{v}^{\top}\! w$. 
Consider an $n$-by-$n$ matrix, say $M$, with complex entries acting on $\mathbb{C}^n$. The matrix preserves the Hermitian product if and only if $\langle Mv,Mw\rangle = \langle v,w\rangle$:
$$\langle Mv, Mw \rangle = \langle v,w\rangle \iff \overline{(Mv)}^{\top}(Mw) = \overline{v}^{\top}w \iff \overline{v}^{\top}(\overline{M}^{\top}\!\! M)w = \overline{v}^{\top}w \, .$$
Thus, $M$ preserves the Hermitian product if and only if $\overline{M}^{\top}\!\! M$ is the $n$-by-$n$ identity matrix, i.e. $M$ is a unitary matrix.
