0
$\begingroup$

Consider the linear regression below:

$$\begin{align}\hat{c}&=\arg\min‖b-Xc‖&(1)\end{align}$$

Where the least squares solution is as follows:

$$\begin{align}\hat{c}&=(X'X)^{-1}X'b&(2)\end{align}$$

It is possible to decompose data matrix as

$$\begin{align}X&=U\Sigma V'&(3)\end{align}$$

In which $U$ and $V$ are unitary matrices and $\Sigma$ is a diagonal matrix which includes the singular values. Putting (3) into (2) leads to

$$\begin{align}\hat{c}&=V\Sigma^{-1}U'b&(4)\end{align}$$

which is common in least squares literature. Now consider the weighted least squares estimator

$$ \begin{align} c&=\arg\min‖W(b-Xc)‖\\ \hat{c}&=(X'WX)^{-1} X'Wb&(5) \end{align} $$

if the same decomposition as (3) is done for (5)

$$ \begin{align} \hat{c}&=(X'WX)^{-1} X'Wb\\ &=(V\Sigma U'WU\Sigma V')^{-1} V\Sigma U'Wb\\ &=(V\Sigma^{-1}U'W^{-1}U\Sigma^{-1}V') V\Sigma U'Wb \\ &=V\Sigma^{-1}U' b \end{align} $$

This is the same as ordinary least squares. In fact the weight matrix $W$ is eliminated. Why does this happen? What is its meaning? If the SVD in (3) exactly holds for all data matrices $X$, then WLS is equivalent with OLS. A possible answer is that (3) is not found exactly for all $X$ matrices. Is that true? However for case where (3) is true, again WLS becomes equal to OLS and this is strange!

$\endgroup$

1 Answer 1

2
$\begingroup$

The wiredness is due to that $X$ is not always invertible, that is, $\Sigma^{-1}$ does not always exist.

Your lines of reasoning is correct, if $X$ is invertible. This may be comprehended alternatively as follows: when $X$ is invertible, there always exists an optimal solution $c^{*} \triangleq X^{-1}b$ such that $Xc^{*} = b$. Consequently, it is also true that $WXc^{*} = Wb$ for all $W$. Therefore, $c^{*}$ is the solution for both LS and WLS.

If $X$ is not invertible, this is not generally true. An exception is that when $W = k I$ for some nonzero constant $k \in \mathbb{C}$, where we weigh everything equally.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .