How to approach specific dimensionality reductions? I'm having difficulty in rationalizing dimensionality reduction (I've used other sources), and I would appreciate it if someone could help me out with a specific example.
Given an $M \times M$ PCA coefficient matrix, how could I reduce an $M \times N$ matrix that has my data into a matrix of desired dimensionality, say $2$ or $3$?
 A: If you mean an $M\times M$ symmetric matrix, then it can be factored as
$$
A = G D G'
$$
where $G$ is an orthogonal matrix (i.e. $G'G=GG'=I_M$) and $D$ is a diagonal matrix.  If $A$ is non-negative-definite (as it would be if I understand your question correctly) then the diagonal entries in $D$, which are eigenvalues, would all be non-negative.
First, replace by $0$ all but the largest two or three eigenvalues.  (The theory is that the small eigenvalues would have been $0$ if not for noise.)  Let's say you retain three of them.  
Those three eigenvalues correspond to eigenvectors that are rows of $G$ (rows rather than columns because you said $M\times N$ rather than $N\times M$). Strip away all other rows of $G$, getting a smaller $3\times M$ matrix $H$.  The matrix $H$ has orthonormal rows but it's not a square matrix and does not have orthonormal columns, so it's not what is normally called an orthogonal matrix. It satisfies $HH'=I_3$ but not $H'H=I_M$.  Then you're replacing the $M\times M$ matrix $A$ with an $M\times M$ matrix of lower rank, specifically of rank $3$:
$$
H\begin{bmatrix} d_1 \\ & d_2 \\ & & d_3 \end{bmatrix} H' = \left( H \begin{bmatrix} \sqrt{d_1} \\ & \sqrt{d_2} \\ & & \sqrt{d_3} \end{bmatrix} \right)\left( H \begin{bmatrix} \sqrt{d_1} \\ & \sqrt{d_2} \\ & & \sqrt{d_3} \end{bmatrix} \right)'.
$$
Let $X$ be your original $M\times N$ data matrix. Normally one would first "adjust" $X$ by subtracting the mean of each row from the row; e.g. if there were $1000$ entries in the first row, one would subtract the mean of those $1000$ numbers from every entry in that row.  The adjusted matrix is the $M\times N$ matrix $XQ$ where $Q$ is the $N\times N$ matrix with $1-\frac 1 N$ in each diagonal entry and $-1/N$ in each off-diagonal entry. ($Q$ is the matrix of an orthogonal projection; it's a symmetric idempotent matrix of rank $N-1$. One of its eigenvalues is $0$; all $N-1$ of the others are $1$.)  Then
$$
\begin{bmatrix} \sqrt{d_1} \\ & \sqrt{d_2} \\ & & \sqrt{d_3} \end{bmatrix} H' X Q
$$
should be the "reduced" $3\times N$ matrix you're looking for.
