I was wondering if I can get some help in getting bounds on the parameters estimated by least squares (LS) and weighted least squares (WLS) methods.

Suppose our observation model is:

$\mathbf{y} = \Phi \mathbf{a} + \boldsymbol{\eta}$

The LS estimator is then given by:

$\hat{\mathbf{a}} = (\Phi^{T}\Phi)^{-1}\Phi^{T}y$

And the WLS estimator is given by:

$\tilde{\mathbf{a}} = (\Phi^{T}W\Phi)^{-1}\Phi^{T}Wy$,

where $W$ is a weight matrix.

We further assume that $\Phi$ is a full rank matrix and $W$ is a positive seimidefinite matrix.

Is there a bound on $||\hat{\mathbf{a}}||$ and $||\tilde{\mathbf{a}}||$ in terms of $||\mathbf{y}||$ and some singular values or norms of $\Phi$ and $W$. What if $W$ is diagonal and positive definite? What if $W$ is diagonal and positive semidefinite definite?


  • $\begingroup$ Can I ask why you're asking in this? I mean where does the answer lead? $\endgroup$ – Michael Jun 27 '13 at 16:50
  • $\begingroup$ Why do you care about the norm of the solution? What about the error? $\endgroup$ – Royi Nov 25 '16 at 12:31

In the OLS case, $\| \hat{\mathbf{a}} \| \leq \| (\Phi^{T}\Phi)^{-1}\| \cdot \| \Phi \| \cdot \|y\|$. The operator norm $\| (\Phi^{T}\Phi)^{-1}\|$ is directly related to the singular values of $\Phi$. Same goes for $\|\Phi \|$ (it is the square root of the largest singular value of $\Phi$).

In the WOLS case, replace $\Phi$ by $W^{\frac{1}{2}} \Phi$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.