I am quite weak when it comes to matrices and would like some help. I am trying to re-derive the weighted least squares below. I am not sure how to go from (2) to (3). How can you just shift the variables around ? I tried transposing the equation but I still get something very different. Also from (4) to (5), how does $wxx^T$ become $X^TWX$ ?

How does the magic happen ? Is there a reference book with all the magical matrix tricks ?

enter image description here


To go from (2) to (3), first note that the rightmost summation is a linear combination of vectors $\tilde{\mathbf{x}}_n$, with coefficients with happen to be scalar products of the form $\beta^T \tilde{\mathbf{x}}_n$. But, this scalar product can also be written as $ \tilde{\mathbf{x}}_n^T \beta$, so the terms become: $w_n (\tilde{\mathbf{x}}_n^T \beta) \tilde{\mathbf{x}}_n$. You can easily check that this really equals $w_n \tilde{\mathbf{x}}_n \tilde{\mathbf{x}}_n^T \beta $, which appears in (3).

Now, the passage from (4) to (5) suggests that $\mathbf{W}$ is a diagonal matrix containing the numbers $w_n$ on its diagonal and also that the vectors $\tilde{\mathbf{x}}_n$ are the columns of $\tilde{\mathbf{X}}^T$. Hence, $\tilde{\mathbf{X}}^T \mathbf{W}$ yields the linear combination $\sum_n w_n \tilde{\mathbf{x}}_n$. If you multiply $\tilde{\mathbf{X}}^T \mathbf{W}$ from the right by $\tilde{\mathbf{X}}$, what you get is a linear combination of outer products of columns of $\tilde{\mathbf{X}}^T$ by rows of $\tilde{\mathbf{X}}$, which are the vectors $\tilde{\mathbf{x}}_n^T$. The coefficients of this linear combination are precisely the numbers $w_n$. Hence, (4) and (5) are really equal.

Notice that it's all a matter of (1) well understanding the definition and the properties of the matrix-matrix and matrix-vector products and (2) breaking down the expression into parts whose meaning is clear for then being able to understand how they interact to produce the overall result.

Hope that helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.