2
$\begingroup$

What are the proper steps to take the derivative with respect to $\alpha$ of

$$\frac 1 2 (x - \alpha g)^T Q (x - \alpha g)$$

to get following?

$$-(x-\alpha g)^T Q g$$

(where $\alpha$ is a scalar, $Q$ is a symmetric positive definite matrix, and $x$ and $g$ are vectors of proper size.)

Background and further explanation:

I saw this differentiation as part of a differentiation of a larger expression in Nocedal and Wright's book Numerical Optimization, just above (3.25). The authors skip explaining differentiation steps and obtain the minimizer $\alpha$, as that is their purpose.

When I try to follow I see that the differentiation of this term must yield as above. Now my problem is, if I were to differentiate the term above I would not know that there would be a transpose there, or $Q$ would need to be in the middle. E.g. if everything were scalars the term would be $\frac 1 2 Q (x - \alpha g)^2$, and using the chain rule, I would obtain the derivative as $-Q(x-\alpha g)g$. Now that the vectors and transposes and matrices are involved, how should one apply the chain rule here properly?

$\endgroup$
2
$\begingroup$

\begin{align} &\frac 1 2 \frac{d}{d\alpha}(x - \alpha g)^T Q (x - \alpha g)\\ =& \frac{1}{2} \frac{d}{d\alpha}\left[x^TQx -\alpha x^TQg-\alpha g^TQx + \alpha^2 g^TQg\right]\\ =& \frac{1}{2} \left[-x^TQg-g^TQx + 2\alpha g^TQg\right]\\ =& \frac{1}{2} \left[-x^TQg-x^TQg + 2\alpha g^TQg\right] \quad \text{using Q is symmetric,}\\ =& \frac{1}{2} \left[-2x^TQg + 2\alpha g^TQg\right]\\ =& -x^TQg + \alpha g^TQg\\ =& -(x- \alpha g)^T Qg\\ \end{align}

Edit: Using the product rule. \begin{align} &\frac 1 2 \frac{d}{d\alpha}(x - \alpha g)^T Q (x - \alpha g)\\ =& \frac{1}{2}\left[ \left(\frac{d}{d\alpha}(x - \alpha g)^T\right)\cdot Q (x - \alpha g)+(x - \alpha g)^T Q\cdot \frac{d}{d\alpha} (x - \alpha g)\right]\\ =& \frac{1}{2} \left[\left(\frac{d}{d\alpha}(x - \alpha g)\right)^T\cdot Q (x - \alpha g)+(x - \alpha g)^T Q\cdot (- g)\right]\\ =& \frac{1}{2} \left[(-g)^T\cdot Q (x - \alpha g)-(x - \alpha g)^T Q g\right] \\ =& \frac{1}{2} \left[-(g)^T\cdot Q (x - \alpha g)-(x - \alpha g)^T Q g\right]\\ =& \frac{1}{2} \left[-(x - \alpha g)^T\cdot Qg -(x - \alpha g)^T Q g\right] \quad \text{using Q is symmetric,}\\ =& -(x- \alpha g)^T Qg\\ \end{align}

$\endgroup$
5
  • $\begingroup$ Would it be possible to do this without doing the multiplication you did in second line? $\endgroup$
    – Zargo
    Nov 21 '17 at 13:21
  • $\begingroup$ @anapprentice As in using a product rule say? Sure. I'll edit now $\endgroup$
    – snulty
    Nov 21 '17 at 13:29
  • $\begingroup$ @anapprentice how's this? $\endgroup$
    – snulty
    Nov 21 '17 at 13:36
  • $\begingroup$ Good, thanks. That trick where you used the fact Q is symmetric was what I was missing, I think. $\endgroup$
    – Zargo
    Nov 21 '17 at 13:49
  • $\begingroup$ @anapprentice No problem :) Yeah $Q$ being symmetric means $Q=Q^T$, and since $x^T Q y$ is supposed to be a number say $r=x^T Q y$, then $r^T=r$, and so $r^T=y^TQ^Tx=y^TQx=x^TQy=r$ $\endgroup$
    – snulty
    Nov 21 '17 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.