2
$\begingroup$

$$f(W)=(Ax-b)^TW(Ax-b)=x^TA^TWAx-2b^TWAx+b^TWb$$

where $f(W)$ is a function of $W$, $A$ is a known matrix, $x$ and $b$ are vectors ($b$ is known). How to get $\frac{\partial f}{\partial W}$?

$\endgroup$
  • $\begingroup$ x and b are fixed, right? If $Ax=b$, then $f(W)=0$ for all $W$. If $Ax \neq b$, then $f(W) is unbounded above and below. Is there more to this problem that you haven't told us? $\endgroup$ – Brian Borchers Feb 13 '17 at 3:19
  • $\begingroup$ @BrianBorchers Sorry I made a mistake understanding the problem. Now I just keep it correct and clean. $\endgroup$ – James Feb 13 '17 at 3:42
  • $\begingroup$ Try $W=\alpha I$ and $W=-\alpha I$ where $\alpha$ is a very large number (say $1 \times 10^{300}$.) $\endgroup$ – Brian Borchers Feb 13 '17 at 4:34
  • $\begingroup$ @BrianBorchers thanks! $\endgroup$ – James Feb 13 '17 at 13:58
2
$\begingroup$

Define the vector $$y=Ax-b$$ and write the function in terms of this new variable and the double-dot (aka Frobenius) product.

In this form, the differential & gradient are easy to calculate $$\eqalign{ f &= yy^T:W \cr df &= yy^T:dW \cr \frac{\partial f}{\partial W} &= yy^T \cr }$$

$\endgroup$
  • $\begingroup$ Should that be $y^Ty$? $\endgroup$ – James Feb 12 '17 at 22:54
  • 2
    $\begingroup$ @James No, $y^Ty$ is a scalar, and the gradient wrt a matrix is a matrix. The same way that the gradient of a scalar function wrt to a vector argument, is a vector and not a scalar. $\endgroup$ – hans Feb 12 '17 at 23:22
1
$\begingroup$

$$f (\mathrm W) := (\mathrm A \mathrm x - \mathrm b)^{\top} \mathrm W (\mathrm A \mathrm x - \mathrm b) = \mbox{tr} \left( (\mathrm A \mathrm x - \mathrm b)(\mathrm A \mathrm x - \mathrm b)^{\top} \mathrm W \right) = \langle (\mathrm A \mathrm x - \mathrm b)(\mathrm A \mathrm x - \mathrm b)^{\top}, \mathrm W \rangle$$

Hence,

$$f ' (\mathrm W) = \color{blue}{(\mathrm A \mathrm x - \mathrm b)(\mathrm A \mathrm x - \mathrm b)^{\top}}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.