I have a question which seems simple, but I would appreciate some comments!
Sometimes models involve a low-rank approximation to a covariance matrix. What confuses me is how you can compute the PDF of these low-rank matrices. As a very simple example, consider the case where:
$\textbf{x} \sim \mathcal{N}(\pmb{\mu}, \Sigma)$
$\textbf{y} = K \textbf{x}$
where $\textbf{x}$ is a vector of length $m$ and $K$ is an $n \times m$ matrix, with $n > m$.
Now by using some standard identities, I believe the distribution of $\textbf{y}$ is:
$\textbf{y} \sim \mathcal{N}(K \pmb{\mu}, K \Sigma K^T)$
That's all well and good, but the covariance matrix
$\Sigma^* = K \Sigma K^T$
is now only of rank $m$, but of size $n \times n$. The multivariate normal PDF involves a term:
$p(\textbf{x}|\mu, \Sigma) \propto \textrm{exp}(-\frac{1}{2} \textbf{x}^T \Sigma^{-1} \textbf{x}) $
However, while everything here seems coherent, the matrix $\Sigma^*$ is not invertible because it is not full rank. I suppose this makes sense; we can't compute the PDF for just any value, because the covariance matrix does not span the space.
I suppose my question is: does this all make sense, and if so, what do people tend to do about this? Does it make sense to use a pseudo-inverse to compute the PDF, and what would that mean?
