# Principal component analysis - covariance matrix

I know that PCA is about rotating the axes of coordinate system so that the covariance matrix of data is diagonal. It means we want to have as much variance in measurement of one type as possible and as little covariance between measurements of different types as possible.

$P$ is a matrix whose rows are the new basis vectors. As it turns out below, it's best to take eigenvectors of our original covariance matrix (expressed in original basis) as the new basis.

First, the most important question:

1) the $i^{th}$ diagonal value of $C_{y}$ is the variance of $X$ along $p_i$. Could you explain why? Are they equal to eigenvalues of the corresponding eigenvectors? Why?

2) Can we always rotate the data points so that covariance matrix of the data is diagonal? It's quite surprising for me that this matrix always exists.

3) PCA is about minimizing redundancy and maximizing variance of measurements of the same type. OK, we can obtain a diagonal covariance matrix - it means all terms of the matrix not on its diagonal are zero, so we minimized them (thus PCA eliminates redundancy completely). But how does it maximize the variance (terms on diagonal)? We don't do anything to maximize them actually...

Source