# Confusing Equality Between Ordinary and Weighted Least Squares After SVD

Consider the linear regression below:

\begin{align}\hat{c}&=\arg\min‖b-Xc‖&(1)\end{align}

Where the least squares solution is as follows:

\begin{align}\hat{c}&=(X'X)^{-1}X'b&(2)\end{align}

It is possible to decompose data matrix as

\begin{align}X&=U\Sigma V'&(3)\end{align}

In which $$U$$ and $$V$$ are unitary matrices and $$\Sigma$$ is a diagonal matrix which includes the singular values. Putting (3) into (2) leads to

\begin{align}\hat{c}&=V\Sigma^{-1}U'b&(4)\end{align}

which is common in least squares literature. Now consider the weighted least squares estimator

\begin{align} c&=\arg\min‖W(b-Xc)‖\\ \hat{c}&=(X'WX)^{-1} X'Wb&(5) \end{align}

if the same decomposition as (3) is done for (5)

\begin{align} \hat{c}&=(X'WX)^{-1} X'Wb\\ &=(V\Sigma U'WU\Sigma V')^{-1} V\Sigma U'Wb\\ &=(V\Sigma^{-1}U'W^{-1}U\Sigma^{-1}V') V\Sigma U'Wb \\ &=V\Sigma^{-1}U' b \end{align}

This is the same as ordinary least squares. In fact the weight matrix $$W$$ is eliminated. Why does this happen? What is its meaning? If the SVD in (3) exactly holds for all data matrices $$X$$, then WLS is equivalent with OLS. A possible answer is that (3) is not found exactly for all $$X$$ matrices. Is that true? However for case where (3) is true, again WLS becomes equal to OLS and this is strange!

The wiredness is due to that $$X$$ is not always invertible, that is, $$\Sigma^{-1}$$ does not always exist.
Your lines of reasoning is correct, if $$X$$ is invertible. This may be comprehended alternatively as follows: when $$X$$ is invertible, there always exists an optimal solution $$c^{*} \triangleq X^{-1}b$$ such that $$Xc^{*} = b$$. Consequently, it is also true that $$WXc^{*} = Wb$$ for all $$W$$. Therefore, $$c^{*}$$ is the solution for both LS and WLS.
If $$X$$ is not invertible, this is not generally true. An exception is that when $$W = k I$$ for some nonzero constant $$k \in \mathbb{C}$$, where we weigh everything equally.