10
$\begingroup$

I want expand an L2-norm with some matrix operation inside.
Assume I have a regression $Y=X\beta+\epsilon$.

I want to solve (meaning expand), $$\displaystyle\|Y-X\beta \|_{2}^2$$ Should I do:
1) $$\displaystyle\|Y\|_{2}^2+2\beta^TX^TX\beta+\|X\|_{2}^2$$ or 2) $$\sum_{i=1}^{k}(y_i-x_i\beta_i)^2$$ $$\sum_{i=1}^{k}(y_i^2-2x_i\beta_i+x_i^2)$$

X: n by k matrix
$\beta$: k by 1 vector
$Y$: n by 1 vector

I think I have seen both: keeping the norm notation as I did in 1), and expand in summation for as I did in 2).
Could someone show me how to do it?

$\endgroup$
5
  • 5
    $\begingroup$ $\|Y - X \beta \|^2 = \|Y\|^2 - 2Y^T X \beta + \| X \beta \|^2 = \|Y\|^2 - 2 Y^T X \beta + \beta^T X^T X \beta$. $\endgroup$
    – littleO
    Jul 8, 2014 at 22:31
  • 1
    $\begingroup$ Fantastic! What class should I take to learn this? None of my class taught me this. $\endgroup$
    – user13985
    Jul 18, 2014 at 0:12
  • 7
    $\begingroup$ I think this is normally part of a linear algebra class, where we learn to work with vectors and matrices. Here are a few useful facts that you can understand right now: if $x$ is a column vector, then $x^T x = \|x\|^2$. If $x$ and $y$ are column vectors, then their dot product $\langle x, y \rangle$ is equal to $x^T y$. Finally, we can use FOIL with column vectors: $(x + y)^T(z + w) = x^T z + x^T w + y^T z + y^T w$. Also, if $A$ and $B$ are matrices, then $(AB)^T = B^T A^T$. These are the rules I used to expand $\|Y - X \beta\|^2$. (I'm assuming our vectors have real number entries.) $\endgroup$
    – littleO
    Jul 18, 2014 at 0:30
  • $\begingroup$ Thanks for breaking it down, it helps very much. I'm glad to accept this as an answer. I have taken linear algebra, the book by Stranger. I have not seen this expansion. $\endgroup$
    – user13985
    Jul 18, 2014 at 0:33
  • 1
    $\begingroup$ I meant Strang, is the author. $\endgroup$
    – user13985
    Jul 18, 2014 at 1:37

2 Answers 2

20
$\begingroup$

The FOIL rule with column vectors $u, v \in \mathbb R^n$ tells us that \begin{align} \|u - v\|^2 &= (u - v)^T (u - v) \\ &= u^T u - u^T v - v^T u + v^T v\\ &= \|u\|^2 - 2 u^T v + \|v\|^2. \end{align} (In the last step, we used the fact that $u^Tv = v^Tu = \sum_{i=1}^n u_iv_i$.)

Applying this to our particular problem, we find that \begin{align} \|Y - X \beta \|^2 &= \|Y\|^2 - 2 Y^T X \beta + \| X \beta \|^2. \end{align}

If we'd like, we could rewrite the final term as \begin{align*} \|X \beta \|^2 &= (X \beta)^T (X \beta) \\ &= \beta^T X^T X \beta. \end{align*}

$\endgroup$
3
  • $\begingroup$ Shouldn't it be $u^Tu - u^Tv - v^Tu + v^Tv$ ? There seem to have been - instead of + in the end. $\endgroup$
    – ares
    Apr 14, 2019 at 14:58
  • $\begingroup$ Furthermore, how is the third term $v^Tu = u^Tv$ ? $\endgroup$
    – ares
    Apr 14, 2019 at 15:15
  • 1
    $\begingroup$ @ares Good catch, I changed the - to +. Regarding $v^T u = u^T v$, both are equal to $\sum_i u_i v_i$, which is the dot product of $u$ and $v$. $\endgroup$
    – littleO
    Apr 14, 2019 at 18:42
0
$\begingroup$

It's hard to know what you're trying to achieve. When possible, avoid coordinates until the end. What do you know here and what do you not know? What do you mean by "solve"? Your first equation would be my preference, dotting $Y-X\beta$ with itself, but you have to get the formula correct.

$\endgroup$
5
  • 1
    $\begingroup$ I want to FOIL it, like $(a-b)^2=a^2-2ab+b^2$ $\endgroup$
    – user13985
    May 14, 2013 at 21:42
  • 1
    $\begingroup$ Yes, so this needs to be $\|Y\|_2^2 - 2Y\cdot X\beta + \|X\beta\|_2^2$. $\endgroup$ May 14, 2013 at 22:04
  • $\begingroup$ How about without the dot? $2Y\cdot X\beta=2Y^TX\beta$? btw, in my first comment, I can write middle term as -2ab or -2ba. But in matrix, one of them would be wrong, what's the rule for this? Do I always count the dimension first? $\endgroup$
    – user13985
    May 14, 2013 at 22:13
  • $\begingroup$ Yes, $Y\cdot W = Y^TW$. Note that in all this we're identifying $1\times 1$ matrices with numbers. So $Y\cdot W = Y^T W = (Y^T W)^T = W^T Y = W\cdot Y$. $\endgroup$ May 15, 2013 at 14:58
  • $\begingroup$ Thanks, figured it out. I also think the answer 1 and 2 up there are correct. $\endgroup$
    – user13985
    May 15, 2013 at 15:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .