Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For the $n\times p$ matrix $\mathbf{X}$, is there any use in approximating $(\mathbf{X}'\mathbf{X})^{-1}$ when $p>>n$? If so, what information might this tell us?

I understand when $p<n$, $\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'$ is the linear projection operator on to the $p$-dimensional subspace. In the $p>>n$ case, this matrix would have to be approximated since it cannot be computed directly. However, I'm not sure how if it supplies any useful information.

share|cite|improve this question

migrated from Nov 17 '12 at 8:41

This question came from our site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

What is p and what is n? – TenaliRaman Nov 17 '12 at 5:36
p is the number of parameters and n is the number of observations – Glen Nov 17 '12 at 5:38
$X^tX$ does not have an inverse: the rank of $X$ (and $X^t$) is at most $n$ ($<p$), and therefore, $X^tX$ has at most rank $n$ (and therefore, isn't full rank). So, you have to be a bit more clear about what you mean by inverting this matrix, to begin with. – user765195 Nov 17 '12 at 7:09

I think your question is somewhat imprecisely phrased: as you say, $X'X$ is not invertible when $p>n$, so $(X'X)^{-1}$ does not exist. Generally speaking, however, $X'X$ is computed as a means to estimate the underlying covariance matrix $\Sigma$ from which the data is generated (for data with mean zero, the sample covariance matrix $S=\frac{1}{n}X'X$ is the maximum likelihood estimator for $\Sigma$).

Along the same lines, it is possible to use $X$ to estimate the inverse covariance matrix $\Sigma^{-1}$, which is well-defined, even though the sample covariance $S$ is not invertible. Knowledge of the inverse covariance matrix is desirable because it describes the partial covariances between variables, i.e. the strength of the relationship between them when all other variables are held constant.

One simple way to estimate $\Sigma^{-1}$ is by adding a multiple of the identity to $S$, i.e. taking $\widehat{\Sigma^{-1}} = (S+\lambda I)^{-1}$ for some constant $\lambda$ such that $S+\lambda I$ is positive definite. Other more complicated methods also exist, and you can find them fairly easily by searching for, say, "inverse covariance estimation". Although this problem was first considered some time ago, many of the most popular methods were only developed recently, so if you're interested in doing a literature search I would focus on more recent sources.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.