1
$\begingroup$

For ordinary least squares (OLS), the solution to the system $X\beta = y$ is

$\hat{\beta} = (X^T X)^{-1} X^T y$

and the variance on the solution parameters is

$Var(\hat{\beta}) = \sigma^2 (X^T X)^{-1}$

where the vector y denoted our observables and $\sigma$ are the errors on these observables.

If I instead would obtained the solution from the weighted least squares as

$\hat{\beta} = (X^T W X)^{-1} X^T W y$

where $W_{ii} = 1 / \sigma^2_i$, what would be the corresponding variance-covariance matrix of $\hat{\beta}$? Is it the same as in the OLS case?

$\endgroup$

1 Answer 1

1
$\begingroup$

In weighted least squares, you have $y=X\beta+\varepsilon$ where $\operatorname E(\varepsilon)=0$ and $\operatorname{Var}(\varepsilon)=\Omega=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_n^2)$ is positive definite.

Weighted least squares estimator of $\beta$ is then $$\hat\beta_{\text{WLS}}=(X^T\Omega^{-1}X)^{-1}X^T\Omega^{-1}y=Py \quad(\text{say})$$

Therefore,

\begin{align} \operatorname{Var}\left(\hat\beta_{\text{WLS}}\right)&= P\cdot\operatorname{Var}(y)\cdot P^T \\&=(X^T\Omega^{-1}X)^{-1}X^T\Omega^{-1}\cdot\operatorname{Var}(y)\cdot((X^T\Omega^{-1}X)^{-1}X^T\Omega^{-1})^T \\&=(X^T\Omega^{-1}X)^{-1}X^T\Omega^{-1}\cdot\Omega\cdot\Omega^{-1}X(X^T\Omega^{-1}X)^{-1} \\&=(X^T\Omega^{-1}X)^{-1} \end{align}

You can see that this matches with $\operatorname{Var}\left(\hat\beta_{\text{OLS}}\right)$ when $\sigma_i^2=\sigma^2$ for each $i$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .