Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am estimating a model minimizing the following objective function,

$ M(\theta) = (Z'G(\theta))'W(Z'G(\theta))$

$Z$ is an $N \times L$ matrix of data, and $W$ is an $L\times L$ weight matrix, neither of which depends on $\theta$. $G(\theta)$ is a function which takes the $K \times 1$ vector of parameters I am estimating into an $N \times 1$ vector of residuals.

I am trying to calculate the Gradient and the Hessian of M to feed to into a Matlab solver ($G(\theta)$ is highly nonlinear). I believe I have the gradient correctly calculated as follows. Let $J = dG/d\theta$ be an $N \times K$ Jacobian matrix, where $J_{i,k}$ is the derivative of element $i$ of $G$ with respect to parameter $k$. Then the gradient vector of M is,

$ \nabla M = 2 (Z'J(\theta))'W (Z'G(\theta))' $

However, I cannot figure out how to take the derivative of this gradient to get the Hessian. I can calculate the vectors of derivatives of each element of $J(\theta)$, but I'm not sure what the order of that derivative matrix should be. I see that I basically need to divide this function up into two parts and use a the product rule, but cannot figure out what the derivative of each part should look like.

Any help would be greatly appreciated. Thank you.

share|cite|improve this question

I was having similar issues with an equation as the second derivative evaluation was not possible.

Numerical Optimization by J Nocedal & S Wright

And The Levenberg-Marquardt Algorithm by Ananth Ranganathan

suggest that the Hessian and Gradient can be approximated rather than directly evaluated. The Hessian can be approximated as the transpose of the Jacobian multiplied by the Jacobian itself. The Gradient can be approximated by the transpose of the Jacobian multiplied by the Residuals.

I hope this is some use to you or points you in a more helpful direction.

share|cite|improve this answer

I assume that $\theta\in\mathbb{R}$ and $W$ is a symmetric matrix (otherwise your formula about $\nabla M$ is false). Moreover $\dfrac{\partial G}{\partial\theta}=J_{i,k}$ where $J_{i,k}=\dfrac{\partial G_{i,k}}{\partial\theta}$. Finally $M(\theta)=(Z^TG)^TWZ^TG$ implies $\dfrac{\partial M}{\partial\theta}=2(Z^TG)^TWZ^T\dfrac{\partial G}{\partial\theta}$ that implies $\dfrac{\partial ^2M}{\partial\theta^2}=2(Z^TG)^TWZ^T\dfrac{\partial ^2G}{\partial\theta^2}+2(Z^T\dfrac{\partial G}{\partial\theta})^TWZ^T\dfrac{\partial G}{\partial\theta}$ and so on. It is a generalization of $(f^2)'=2ff',(f^2)"=2(f')^2+2ff"$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.