Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am having trouble try to show that this linear regression summation:

$J(w) = \sum^m_{i=1} u_i (w^T x_i - y_i)^2$

can be rewritten in the following matrix form:

$J(w) = (Xw - y)^T U(Xw - y)$

Apparently it's possible and I do not know where the $U$ comes from. Any help would be appreciated.

share|cite|improve this question
up vote 1 down vote accepted

The subscript denotes the $i$'th columns of the corresponding matrices. Rewrite $J(w)$ as $$ \sum^m_{i=1} (w^T x_i - y_i) u_i (w^T x_i - y_i) $$ Then, notice that $w^Tx_i = x_i^Tw$ is a scalar quantity (dot product of two vectors), so $$ \sum^m_{i=1} (x_i^Tw - y_i) u_i (x_i^Tw - y_i) $$ Notice that $u_i$ is also scalar which hints us for a diagonal matrix $U$. The structure then becomes, $$ \pmatrix{x_1^Tw-y_1 &\ldots &x_m^Tw-y_m}\pmatrix{u_1 \\&u_2\\&&\ddots\\&&&u_m}\pmatrix{x_1^Tw-y_1 \\\vdots \\x_m^Tw-y_m} $$ With a final touch, we almost arrive at the result with a typo. $$ J(w) = (X^Tw-y)^T U(X^Tw-y) $$

share|cite|improve this answer
The typo is the transpose of $(X^T w - y)^T$ ? – gprime Sep 23 '11 at 12:12
also how did you turn $(X^T_i w - y_i)$ into a row and column vector? – gprime Sep 23 '11 at 12:14
it is an element wise operation to compute the first element of $X^Tw$ you simply take the first row of $X^T$ which is denoted by $x_i^T$. I hope it is slightly clearer now> – user13838 Sep 23 '11 at 12:30
Thanks, this helped me a lot – gprime Sep 28 '11 at 11:40
@gprime You are very welcome – user13838 Sep 28 '11 at 11:41

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.