# $(X^tX)^{-1}$ when $p>>n$

For the $n\times p$ matrix $\mathbf{X}$, is there any use in approximating $(\mathbf{X}'\mathbf{X})^{-1}$ when $p>>n$? If so, what information might this tell us?

I understand when $p<n$, $\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'$ is the linear projection operator on to the $p$-dimensional subspace. In the $p>>n$ case, this matrix would have to be approximated since it cannot be computed directly. However, I'm not sure how if it supplies any useful information.

-

## migrated from stats.stackexchange.comNov 17 '12 at 8:41

This question came from our site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

What is p and what is n? – TenaliRaman Nov 17 '12 at 5:36
p is the number of parameters and n is the number of observations – Glen Nov 17 '12 at 5:38
$X^tX$ does not have an inverse: the rank of $X$ (and $X^t$) is at most $n$ ($<p$), and therefore, $X^tX$ has at most rank $n$ (and therefore, isn't full rank). So, you have to be a bit more clear about what you mean by inverting this matrix, to begin with. – user765195 Nov 17 '12 at 7:09

I think your question is somewhat imprecisely phrased: as you say, $X'X$ is not invertible when $p>n$, so $(X'X)^{-1}$ does not exist. Generally speaking, however, $X'X$ is computed as a means to estimate the underlying covariance matrix $\Sigma$ from which the data is generated (for data with mean zero, the sample covariance matrix $S=\frac{1}{n}X'X$ is the maximum likelihood estimator for $\Sigma$).
Along the same lines, it is possible to use $X$ to estimate the inverse covariance matrix $\Sigma^{-1}$, which is well-defined, even though the sample covariance $S$ is not invertible. Knowledge of the inverse covariance matrix is desirable because it describes the partial covariances between variables, i.e. the strength of the relationship between them when all other variables are held constant.
One simple way to estimate $\Sigma^{-1}$ is by adding a multiple of the identity to $S$, i.e. taking $\widehat{\Sigma^{-1}} = (S+\lambda I)^{-1}$ for some constant $\lambda$ such that $S+\lambda I$ is positive definite. Other more complicated methods also exist, and you can find them fairly easily by searching for, say, "inverse covariance estimation". Although this problem was first considered some time ago, many of the most popular methods were only developed recently, so if you're interested in doing a literature search I would focus on more recent sources.