Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am new to linear algebra and I have the following doubts:

  1. In weighted least square estimation of the system $Ax = b$ we minimize the weighted value of the error $e = b - Ax$ and the best $\hat{x}$ is given by $( A^T \Sigma^{-1}A )^{-1} A^T\Sigma^{-1} b$ where $\Sigma$ is the covariane matrix of the error $e$. Why is the covariance matrix $\Sigma{e}$ the best choice for the weighting matrix? Is there any derivation for it? Please refer me to its link or sum hints will also do.

  2. For the same linear system $e = b - Ax$ is $E(ee^T) = E(bb^T)$ given that error is unbiased (i.e. $E(e) = 0$)?

share|cite|improve this question

This is a very basic and important topic. Regarding the deterministic and stochastic least square estimation, I highly recommend the book "Linear estimation" by Thomas Kailath.

First, I think we need to clarify the problem statement. The system here should be $y=Ax+v$ where $y$ is the measurement (I prefer to use $y$ instead of $b$), and $v$ is a zero-mean random noise who is uncorrelated to $x$. In this system, $x$ is deterministic and $v$ and $y$ are stochastic.

Second, the estimator $\hat{x}=Ky$ with $K=(A^T\Sigma^{-1}A)^{-1}A^T\Sigma^{-1}$ is a linear minimum variance unbiased estimator of $x$. Note $\mathbb{E}(v)=0$ it is easy to check $\mathbb{E}(\hat{x})=x$ and $\mathbb{E}{(x-\hat{x})(x-\hat{x})^T}=(A^T\Sigma^{-1}A)^{-1}$. The proof that the estimation variance is the minimum can be found on page 97 of "Linear estimation".

Third, I think $\mathbb{E}(ee^T)=K\Sigma K^T$ instead of $\mathbb{E}(ee^T)=\Sigma$. Hint: $KA=I$ because $\mathbb{E}\hat{x}=KAx=x$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.