I have proven, as below, that if $X \in M_{n \times n}$ is symmetric, then it has a SVD.
$D(\lambda_i) = \text{Diag}(\lambda_i)$ is a diagonal matrix with entries $\lambda_1, \lambda_2, \dots$.
$X^{\prime}$ denotes the transpose of $X$.
We say a square matrix $P$ is orthogonal if $P^{\prime} = P^{-1}$.
Theorem. Suppose $A \in M_{n \times n}$ is symmetric. Then there exists an orthogonal matrix $P$ such that $P^{\prime}AP = \text{Diag}(\lambda_i)$, where $\lambda_1, \lambda_2, \dots, \lambda_n$ are the eigenvalues of $A$.
Let $\{v_i\}_{i \in \{1, 2, \dots, n\}}$ be an orthonormal set of eigenvectors, each vector corresponding to an element in $\{\lambda_i\}_{i \in \{1, 2, \dots, n\}}$.
Let $P = \begin{bmatrix} v_1 & \cdots & v_n \end{bmatrix}$. Then \begin{align*} P^{\prime}AP = \begin{bmatrix} v^{\prime}_1 \\ \vdots \\ v^{\prime}_n \end{bmatrix}\begin{bmatrix} Av_1 & \cdots & Av_n \end{bmatrix} &= \begin{bmatrix} v^{\prime}_1 \\ \vdots \\ v^{\prime}_n \end{bmatrix}\begin{bmatrix} \lambda_1v_1 & \cdots & \lambda_nv_n \end{bmatrix} \\ &= \begin{bmatrix} \lambda_1 v^{\prime}_1 v_1 & \cdots & \lambda_n v^{\prime}_1 v_n \\ \vdots & \ddots & \vdots \\ \lambda_1 v^{\prime}_n v_1 & \cdots & \lambda_n v^{\prime}_n v_n \end{bmatrix}\\ &= \begin{bmatrix} \lambda_1 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & \lambda_n \end{bmatrix} \\ &= \text{Diag}(\lambda_i) \end{align*} since \begin{equation*} v^{\prime}_iv_j = v_i \cdot v_j = \begin{cases} 1, & i = j \\ 0, & i \neq j \end{cases} \end{equation*} due to orthonormality.
Corollary. Suppose $A \in M_{n \times n}$ is symmetric. Then, by Theorem, $A = PD(\lambda_i)P^{\prime}$. This is known as the singular value decomposition of $A$.
Since $P$ is orthogonal, it is invertible. Furthermore, $$ A = \left(P^{\prime}\right)^{-1}D(\lambda_i)P^{-1} = \left(P^{-1}\right)^{-1}D(\lambda_i)P^{\prime} = PD(\lambda_i)P^{\prime}\text{.} $$
Does this apply for arbitrary matrices (i.e., not necessarily symmetric nor square matrices)? I came up with this question after looking at this solution which is reliant on $X \in M_{n \times p}$ having a SVD.
I have seen this Wikipedia page which says that $A = B\text{Diag}(\text{something})C$, where $B$ is orthogonal and $\text{Diag}(\text{something})$ is a diagonal matrix (but does the diagonal necessarily have to consist of the eigenvalues of $A$?) and I have no idea what $C$, this "conjugate transpose" is.
Could someone help clarify the concept of a SVD for an arbitrary matrix and perhaps help me with a sketch of a proof?