What is the difference between diagonalization and orthogonal diagonalization? I am confused about the following.
When you diagonalize a $n\times n$ matrix $A$, you write $A$ as $PDP^{-1}$ with $P$ being orthogonal. Because if $P$ wasn't orthogonal, it wouldn't be invertable.
Then why don't we call this "orthogonal diagonalization"?
When you diagonalize a $n\times n$ symmetric matrix $A$ (so $A = A^T$), you write $A$ as $PDP^T$, because $P^{-1}= P^T$.
But if $P^{-1}= P^T$, doesn't that imply that $P^TP=I$ and thus that P is orthonormal?
Then why don't we call this "orthonormal diagonalization"?
 A: If $A$ is diagonalizable, we can write $A=S \Lambda S^{-1}$, where $\Lambda$ is diagonal. Note that $S$ need not be orthogonal. Orthogonal means that the inverse is equal to the transpose. A matrix can very well be invertible and still not be orthogonal, but every orthogonal matrix is invertible. Now every symmetric matrix is orthogonally diagonalizable, i.e. there exists orthogonal matrix $O$ such that $A=O \Lambda O^T$. It might help to think of the set of orthogonally diagonalizable matrices as a proper subset of the set of diagonalizable matrices.
A: Being diagonalizable does not imply that it can be diagonalized with an orthogonal matrix.
The relevant result is: A matrix is unitarily diagonalizable iff it is normal (ie, $A^* A = A A^*$).
For example, $A = \begin{bmatrix} 1 & 1 \\ 0 & 2 \end{bmatrix}$. It is straightforward to check that $A$ is not normal, has two distinct eigenvalues, and the eigenspaces are $\mathbb{sp} \{ (1,0)^T \}$ ($\lambda=1$) and $\mathbb{sp} \{ (1,1)^T \}$ ($\lambda=2$) respectively.
It is easy to see that the eigenspaces are not orthogonal and that $A$ can be diagonalized by taking any non-zero vector from the two eigenspaces, say $p_1,p_2$, forming the matrix $P = \begin{bmatrix} p_1 & p_2 \end{bmatrix}$.
Then you will have $A P = P \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}$, and $P$ is invertible (but not orthogonal) because $p_1,p_2$ are linearly independent.
Note: Hermitian matrices (or symmetric in the real case) are 'automatically' normal and can always be unitarily (orthogonally) diagonalized.
Note: Any orthogonal $U$ matrix can be 'turned into' an orthonormal matrix $\tilde{U}$ in the following way: Let $\Lambda = U^* U$, then $\Lambda$ is diagonal with positive entries on the diagonal. Hence we can define the square root $\sqrt{\Lambda}$ as the diagonal matrix of corresponding square roots. Then $\tilde{U} = U \sqrt{\Lambda}$ is orthonormal.
A: A matrix $P$ is called orthogonal if $P^{-1} = P^T$. Thus the first statement is just diagonalization while the one with $PDP^T$ is actually the exact same statement as the first one, but in the second case the matrix $P$ happens to be orthogonal, hence the term "orthogonal diagonalization". 
It's all a matter of getting the definitions to coincide. If you wonder why the latter matrix is called orthogonal and not orthonormal, then this is a little deeper question.
Hope that helps,
