Deep learning Book, page 228

...We will further simplify the analysis by making a quadratic approximation to the objective function in the neighborhood of the value of the weights that obtains minimal unregularized training cost,$$w^∗=arg min_{w}J(w).$$ If the objective function is truly quadratic, as in the case of fitting a linear regression model with mean squared error, then the approximation is perfect. The approximation J ˆ is given by:

$$ (1) \,\,\,\,\,\,\hat{\,\,\,\,\,J(\theta)} = J(w^*) + \frac{1}{2} (w-w^*)^T H (w-w^*) $$

Some objective function can be approximated in the neighborhood of optimal values w* with Taylor series (this is a regularization expression, w are parameters of this optimization).

From here on they take derivative w.r.t w:

...where H is the Hessian matrix of J with respect to w evaluated at w ∗ . There is no first-order term in this quadratic approximation, because w ∗ is defined to be a minimum, where the gradient vanishes. Likewise, because w ∗ is the location of a minimum of J , we can conclude that H is positive semidefinite. The minimum of J ˆ occurs where its gradient is:

$$ (2)\,\,\,\,\,\, \nabla_{w} \hat{\,\,J(w)} = H (w-w^*)$$

My question. I tried to produce this result using derivative rules: $$ \frac{\partial{x^T H x}}{\partial{x}} = x^T(H +H^T) $$ $$ \frac{\partial{b^T H x}}{\partial{x}} = b^TH $$

\begin{equation} (3) \,\,\,\,\,\,\nabla_{w} \hat{\,\,J(w)} = \frac{1}{2}(w^THw - w^TH(w^*) - (w^*)^THw + (w^*)^TH(w^*)) \end{equation}

1.) How do i get the gradient term for $ \nabla_{w}\hat{J(w)}$
2.) Why do authors conclude H must be positive semidefinite.
3.) In here i found that if assumed W is symmetric: $$ \frac{\partial}{\partial{x}}(x-s)^TW(x-s) = W(x-s) $$ This expression is the same as in (2). So does being positive semidefinite imply symmetricity?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.