1
$\begingroup$

I am learning about linear regression, and the goal is to find parameters $\beta$, that minimize the RSS. My textbook accomplishes this by finding $\partial \text{ RSS} /\partial \beta = 0$ However, I am slightly stuck on the following step:

They define:

$RSS(\beta) = (\mathbf{y} - \mathbf{X}\beta)^{T} (\mathbf{y}-\mathbf{X}\beta$,

where $\beta$ are scalars, $y$ is a column vector, and $X$ is a matrix.

They find that

$\frac{\partial RSS}{\partial \beta} = -2\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)$

I tried deriving this result. I first wrote: $(\mathbf{y} - \mathbf{X}\beta)^{T} (\mathbf{y}-\mathbf{X}\beta) = (\mathbf{y}^{T} - \mathbf{X}^{T}\beta)(\mathbf{y} - \mathbf{X}\beta)$

I then expanded the two terms in brackets: $\mathbf{y}^{T}\mathbf{y} - \mathbf{y}^{T}\mathbf{X}\beta - \mathbf{y}\mathbf{X}^{T}\beta + \mathbf{X}^{T}\mathbf{X}\beta^2$

Now, I differentiate this with respect to $\beta$: $-\mathbf{y}^{T}\mathbf{X} - \mathbf{y}\mathbf{X}^{T} + 2\beta \mathbf{X}^{T}\mathbf{X}$

This is where I get stuck, comparing my result with the derived result, we both have the $2\beta \mathbf{X}^{T}\mathbf{X}$ term, but I don't know how my first 2 terms should simplify to give $-2\mathbf{X}^{T}\mathbf{y}$.

$\endgroup$
  • 1
    $\begingroup$ Did you appreciate any of the answers? You should accept the best answer to mark this question as answered. $\endgroup$ – LinAlg Jul 31 '18 at 19:46
2
$\begingroup$

Note that $\beta$ is not a scalar, but a vector.

Let $$\mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{bmatrix}$$ $$\mathbf{X} = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1p} \\ x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{Np} \end{bmatrix}$$ and $$\beta = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_p \end{bmatrix}\text{.}$$ Then $\mathbf{X}\beta \in \mathbb{R}^N$ and $$\mathbf{X}\beta = \begin{bmatrix} \sum_{j=1}^{p}b_jx_{1j} \\ \sum_{j=1}^{p}b_jx_{2j} \\ \vdots \\ \sum_{j=1}^{p}b_jx_{Nj} \end{bmatrix} \implies \mathbf{y}-\mathbf{X}\beta=\begin{bmatrix} y_1 - \sum_{j=1}^{p}b_jx_{1j} \\ y_2 - \sum_{j=1}^{p}b_jx_{2j} \\ \vdots \\ y_N - \sum_{j=1}^{p}b_jx_{Nj} \end{bmatrix} \text{.}$$ Therefore, $$(\mathbf{y}-\mathbf{X}\beta)^{T}(\mathbf{y}-\mathbf{X}\beta) = \|\mathbf{y}-\mathbf{X}\beta \|^2 = \sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)^2\text{.} $$ We have, for each $k = 1, \dots, p$, $$\dfrac{\partial \text{RSS}}{\partial b_k} = 2\sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)(-x_{ik}) = -2\sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{ik}\text{.}$$ Then $$\begin{align}\dfrac{\partial \text{RSS}}{\partial \beta} &= \begin{bmatrix} \dfrac{\partial \text{RSS}}{\partial b_1} \\ \dfrac{\partial \text{RSS}}{\partial b_2} \\ \vdots \\ \dfrac{\partial \text{RSS}}{\partial b_p} \end{bmatrix} \\ &= \begin{bmatrix} -2\sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{i1} \\ -2\sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{i2} \\ \vdots \\ -2\sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{ip} \end{bmatrix} \\ &= -2\begin{bmatrix} \sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{i1} \\ \sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{i2} \\ \vdots \\ \sum_{i=1}^{N}\left(y_i-\sum_{j=1}^{p}b_jx_{ij}\right)x_{ip} \end{bmatrix} \\ &= -2\mathbf{X}^{T}(\mathbf{y}-\mathbf{X}\beta)\text{.} \end{align}$$

$\endgroup$
  • $\begingroup$ Hi. Thanks for this very detailed derivation! Just a quick question: you have denoted as $x_{ip}$, above, which would mean that the transpose is $x_{pi}$, but in your final line, you don't have $x_{pi}$, but $x_{ip}$, so wondering where the $X^{T}$ comes from in the last equality. $\endgroup$ – Thomas Moore Aug 1 '18 at 16:09
  • 1
    $\begingroup$ @ThomasMoore I would recommend that you do the multiplication yourself to see that the above is true, but at a very simplistic level: remember that all you're doing when you're doing matrix multiplication is dot products of rows of the first matrix to columns of the second matrix. The $p$th column of $\mathbf{X}$ ends up being the $p$th row of $\mathbf{X}^T$ when you're doing the transposition. Thus, when you're performing a dot product between the $p$th row of $\mathbf{X}^{T}$ and $\mathbf{y} - \mathbf{X}\beta$, ultimately, you're using all of the entries of the $p$th column of $\mathbf{X}$. $\endgroup$ – Clarinetist Aug 1 '18 at 16:28
1
$\begingroup$

The correct transpose (see property 3) is $(\mathbf{y} - \mathbf{X}\beta)^{T} (\mathbf{y}-\mathbf{X}\beta) = (\mathbf{y}^{T} - \beta^T\mathbf{X}^{T})(\mathbf{y} - \mathbf{X}\beta)$

The correct expansion is $\mathbf{y}^{T}\mathbf{y} - \mathbf{y}^{T}\mathbf{X}\beta - \beta^T \mathbf{X}^{T} \mathbf{y} + \beta^T\mathbf{X}^{T}\mathbf{X}\beta$

You can simplify the expansion to: $$\mathbf{y}^{T}\mathbf{y} + (-\mathbf{X}^{T} \mathbf{y})^T \beta + (-\mathbf{X}^{T} \mathbf{y})^T \beta + \beta^T\mathbf{X}^{T}\mathbf{X}\beta$$ And the result readily follows.

$\endgroup$
  • $\begingroup$ Hi. Thanks for this. But since $\beta$ is a scalar, isn't $\beta^T = \beta$? $\endgroup$ – Thomas Moore Jul 31 '18 at 15:49
  • 1
    $\begingroup$ @ThomasMoore my derivation applies both to scalars and to vectors; if you focus on scalars, your derivation goes wrong where you write $yX^T$ in the expansion: that is a matrix and should be $X^T y$. You can then use that $X^Ty = y^T X$. $\endgroup$ – LinAlg Jul 31 '18 at 15:52
1
$\begingroup$

Expand the brackets to write $$ \begin{align} RSS(\beta)&=y'y-y'X\beta-\beta'X'y+\beta'X'X\beta\\ &=y'y-2\beta'X'y+\beta'X'X\beta \end{align} $$ where primes denote the transpose and $y'X\beta=\beta'X'y= (y'X\beta)'$ since $y'X\beta$ is a $1\times 1$ vector. Now we can differentiate to get that $$ \frac{\partial RSS(\beta)}{\partial \beta}=-2X'y+2X'X\beta=-2X'(y-X\beta) $$ Here we used two properties. First, if $u=\alpha'x$ where $\alpha,x\in\mathbb{R}^n$, then $$ \frac{\partial u}{\partial x_j}=\alpha_j\implies \frac{\partial u}{\partial x}=\alpha. $$ One should notice that $\frac{\partial u}{\partial x}$ in this case represents the gradient. Second if $u=x'Ax=\sum_{i=1}^n\sum_{j=1}^na_{ij} x_{i} x_{j}$ where $A\in M_{n\times n}(\mathbb{R}) $ and $x\in\mathbb{R^n}$, then $$ \frac{\partial u}{\partial x_{\ell}}=\sum_{i=1}^na_{i\ell}x_{i}+\sum_{i=1}^na_{\ell i}x_{i}=[(A'+A)x]_{\ell} \implies \frac{\partial u}{\partial x}=(A'+A)x. $$ In particular if $A$ is symmetric (like $X'X$ as above), we have that $\frac{\partial u}{\partial x}=2Ax$

$\endgroup$
1
$\begingroup$

Remark: $\beta$ is a vector.

In multiple regression, if you have $n$ independent variables, therefore you have $n+1$ parameters to estimate (included intercept), that is: $$y_t=\beta_0+\beta_1X_{1t}+...\beta_nX_{nt}+e_{t},$$ where each $\beta_{i}$ is scalar. We can write aforementioned with matrix notation (your problem is in matrix notation): $$y=X\beta+e,$$ where $X$ is matrix, $y,\beta$ and $e$ are vectors! More precisely, $\beta_{i}$ is scalar, but $\beta$ is vector. Furthermore, you can note that unique solution of the problem that you have mentioned is the following: $$\beta=(X^{T}X)^{-1}X^{T}y,$$ where you can note easily that $\beta$ is a vector.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.