How can I get the gradient of the normal equation for weighted linear regression?

Question

The normal equation for weighted linear regression looks like this:

$$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\in\Re^{m\times m}$, and $W$ is a diagonal matrix, with the diagonals being the $m$ different weights. As a point of reference, this can also be written as

$$J(\theta) = \frac{1}{2}\sum_{i=1}^mw^{(i)}(\theta^Tx^{(i)} - y^{(i)}),$$ where the superscripts indicate the $i$-th element of that vector or diagonal.

I am trying to use the normal equation to show solve for $\theta$, specifically by taking the gradient of the normal equation and setting it equal to zero.

My attempt

I begin by expanding the equation:

\begin{equation} \begin{split} J(\theta) & = (\theta^TX^T - y^T)(WX\theta - Wy) \\ & = \theta^TX^TWX\theta - \theta^TX^TWy - y^TWX\theta + y^TWy \end{split} \end{equation}

Now I know that the answer to my problem is $$\nabla_{\theta}J(\theta) = 2(X^TWX\theta - X^TWy),$$

and the only way I can see to achieve this from what I have is if $$J(\theta) = \theta^TX^TWX\theta -2y^TWX\theta + y^TWy.$$ From my initial expansion of $J(\theta)$, this would imply that $$\theta^TX^TWy = y^TWX\theta.$$

I can't see how this last equality could hold.

I'm quite new to linear algebra, so there is the possibility that I'm making some cardinal error. If so, could someone please outline where my line of reasoning here breaks down? If not, could someone please outline how the last equality holds?

Answer 1

$$X^TWy$$ is a $1\times1$ matrix so that $$(\theta^T(X^TWy))^T = (y^TW^TX)\theta=(y^TWX)\theta$$ as $W$ is symmetric.

Answer 2

$$ J(\beta) = \| W^{1/2} X \beta - W ^{1/2} y \|_2^2 $$

$$ \nabla J(\beta)=2X'W^{1/2}( W^{1/2} X \beta - W ^{1/2} y)\propto X'WX\beta-X'Wy=0, $$ $$ \to\beta = (X'WX)^{-1}X'Wy, $$ $$ \frac{\partial}{\partial \beta \beta ^ T}J(\beta) = 2X'WX \to \text{positive semi-definite matrix} $$