0
$\begingroup$

Deep learning Book, page 228

...We will further simplify the analysis by making a quadratic approximation to the objective function in the neighborhood of the value of the weights that obtains minimal unregularized training cost,$$w^∗=arg min_{w}J(w).$$ If the objective function is truly quadratic, as in the case of fitting a linear regression model with mean squared error, then the approximation is perfect. The approximation J ˆ is given by:

$$ (1) \,\,\,\,\,\,\hat{\,\,\,\,\,J(\theta)} = J(w^*) + \frac{1}{2} (w-w^*)^T H (w-w^*) $$

Some objective function can be approximated in the neighborhood of optimal values w* with Taylor series (this is a regularization expression, w are parameters of this optimization).

From here on they take derivative w.r.t w:

...where H is the Hessian matrix of J with respect to w evaluated at w ∗ . There is no first-order term in this quadratic approximation, because w ∗ is defined to be a minimum, where the gradient vanishes. Likewise, because w ∗ is the location of a minimum of J , we can conclude that H is positive semidefinite. The minimum of J ˆ occurs where its gradient is:

$$ (2)\,\,\,\,\,\, \nabla_{w} \hat{\,\,J(w)} = H (w-w^*)$$

My question. I tried to produce this result using derivative rules: $$ \frac{\partial{x^T H x}}{\partial{x}} = x^T(H +H^T) $$ $$ \frac{\partial{b^T H x}}{\partial{x}} = b^TH $$

\begin{equation} (3) \,\,\,\,\,\,\nabla_{w} \hat{\,\,J(w)} = \frac{1}{2}(w^THw - w^TH(w^*) - (w^*)^THw + (w^*)^TH(w^*)) \end{equation}

1.) How do i get the gradient term for $ \nabla_{w}\hat{J(w)}$
2.) Why do authors conclude H must be positive semidefinite.
3.) In here i found that if assumed W is symmetric: $$ \frac{\partial}{\partial{x}}(x-s)^TW(x-s) = W(x-s) $$ This expression is the same as in (2). So does being positive semidefinite imply symmetricity?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.