The normal equation for weighted linear regression looks like this:

$$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\in\Re^{m\times m}$, and $W$ is a diagonal matrix, with the diagonals being the $m$ different weights. As a point of reference, this can also be written as

$$J(\theta) = \frac{1}{2}\sum_{i=1}^mw^{(i)}(\theta^Tx^{(i)} - y^{(i)}),$$ where the superscripts indicate the $i$-th element of that vector or diagonal.

I am trying to use the normal equation to show solve for $\theta$, specifically by taking the gradient of the normal equation and setting it equal to zero.

My attempt

I begin by expanding the equation:

\begin{equation} \begin{split} J(\theta) & = (\theta^TX^T - y^T)(WX\theta - Wy) \\ & = \theta^TX^TWX\theta - \theta^TX^TWy - y^TWX\theta + y^TWy \end{split} \end{equation}

Now I know that the answer to my problem is $$\nabla_{\theta}J(\theta) = 2(X^TWX\theta - X^TWy),$$

and the only way I can see to achieve this from what I have is if $$J(\theta) = \theta^TX^TWX\theta -2y^TWX\theta + y^TWy.$$ From my initial expansion of $J(\theta)$, this would imply that $$\theta^TX^TWy = y^TWX\theta.$$

I can't see how this last equality could hold.

I'm quite new to linear algebra, so there is the possibility that I'm making some cardinal error. If so, could someone please outline where my line of reasoning here breaks down? If not, could someone please outline how the last equality holds?


2 Answers 2


$$X^TWy$$ is a $1\times1$ matrix so that $$(\theta^T(X^TWy))^T = (y^TW^TX)\theta=(y^TWX)\theta$$ as $W$ is symmetric.


$$ J(\beta) = \| W^{1/2} X \beta - W ^{1/2} y \|_2^2 $$

$$ \nabla J(\beta)=2X'W^{1/2}( W^{1/2} X \beta - W ^{1/2} y)\propto X'WX\beta-X'Wy=0, $$ $$ \to\beta = (X'WX)^{-1}X'Wy, $$ $$ \frac{\partial}{\partial \beta \beta ^ T}J(\beta) = 2X'WX \to \text{positive semi-definite matrix} $$

  • $\begingroup$ There is a $W$ missing, see en.wikipedia.org/wiki/Least_squares#Weighted_least_squares $\endgroup$ May 23, 2018 at 9:19
  • $\begingroup$ Sure. Corrected. Thanks $\endgroup$
    – V. Vancak
    May 23, 2018 at 9:20
  • $\begingroup$ @V.Vancak Thanks for the answer. But shouldn't it be: $J(\theta)^{1/2}=||W^{1/2}X\beta - W^{1/2}y||_2$, because of the power of $1/2$ for the L2-norm? $\endgroup$
    – quanty
    May 23, 2018 at 13:20
  • $\begingroup$ @quanty You are right. I've added the square (However if will not change the solution as this is monotone transformation of the $\mathcal{l}_2$ norm risk function). $\endgroup$
    – V. Vancak
    May 23, 2018 at 15:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .