Consider the linear regression model:

$$y = X\beta + e\\E[e] = 0 \quad E[ee^T] = V$$

It is well known that the minimum variance affine unbiased estimator (MVAUE) of $\beta$ exists if an only if $X$ has linearly independent columns. In this case, the MVAUE is unique

$$\hat\beta = (X^T V_0^+ X)^+ X^T V_0^+y$$

where $V_0 := V + XX^T$ and the superscript "+" denotes the Moore-Penrose inverse. Moreover, $\hat\beta$ is the solution to the following linear least squares problem:

$$\hat \beta = \arg \min_\beta \, (y - X\beta)^T V_0^+ (y - X\beta)$$

It is known that this can be generalized slightly. If $U$ is a positive semidefinite matrix such that $\mathrm{col}\, X \subseteq \mathrm{col}\, V_1$, where $V_1 := V + XUX^T$, then

$$\hat \beta = (X^T V_1^+ X)^+ X^T V_1^+y = \arg \min_\beta \, (y - X\beta)^T V_1^+ (y - X\beta)$$

I am curious if there is a direct proof for the equivalence of the two forms. In other words, I want to know if there is a straightforward way to show

$$(X^T V_0^+ X)^+ X^T V_0^+ = (X^T V_1^+ X)^+ X^T V_1^+$$

provided, of course, that $U$ satisfies the aforementioned condition. Some back-of-envelope calculations I did in Octave suggest that this equation actually holds regardless of whether or not $X$ has linearly independent columns.

Chapter 4 (in particular, section i) of C. R. Rao's "Linear Statistical Inference and it's Applications" and chapter 13 of Magnus and Neudecker's "Matrix Differential Calculus with Applications in Statistics and Econometrics" are two good references on the subject, for the curious.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.