$(X^tX)^{-1}$ when $p>>n$ For the $n\times p$ matrix $\mathbf{X}$, is there any use in approximating $(\mathbf{X}'\mathbf{X})^{-1}$ when $p>>n$?  If so, what information might this tell us?
I understand when $p<n$, $\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'$ is the linear projection operator on to the $p$-dimensional subspace.  In the $p>>n$ case, this matrix would have to be approximated since it cannot be computed directly. However, I'm not sure how if it supplies any useful information.  
 A: I think your question is somewhat imprecisely phrased: as you say, $X'X$ is not invertible when $p>n$, so $(X'X)^{-1}$ does not exist. Generally speaking, however, $X'X$ is computed as a means to estimate the underlying covariance matrix $\Sigma$ from which the data is generated (for data with mean zero, the sample covariance matrix $S=\frac{1}{n}X'X$ is the maximum likelihood estimator for $\Sigma$). 
Along the same lines, it is possible to use $X$ to estimate the inverse covariance matrix $\Sigma^{-1}$, which is well-defined, even though the sample covariance $S$ is not invertible. Knowledge of the inverse covariance matrix is desirable because it describes the partial covariances between variables, i.e. the strength of the relationship between them when all other variables are held constant.
One simple way to estimate $\Sigma^{-1}$ is by adding a multiple of the identity to $S$, i.e. taking $\widehat{\Sigma^{-1}} = (S+\lambda I)^{-1}$ for some constant $\lambda$ such that $S+\lambda I$ is positive definite. Other more complicated methods also exist, and you can find them fairly easily by searching for, say, "inverse covariance estimation". Although this problem was first considered some time ago, many of the most popular methods were only developed recently, so if you're interested in doing a literature search I would focus on more recent sources.
