Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

According to the wikipedia article:


$S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2$

Taking the derivative with respect to δ and setting the result to zero gives:

$(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)])$


My attempt to derive the equation:

$\|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2 = (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

using product rule:

$\frac{\partial \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2}{\partial \boldsymbol\delta} = (-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta) + (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

The dimensions of the left and right side don't match. I believe there might be something wrong with my differentiation. There seems to be a transpose missing, but I'm not sure what would cause a transpose in the differentiation operation.

share|cite|improve this question
You did have a look at that comment I left in your MO question, yes? – J. M. Nov 16 '10 at 13:29
Yes, I had a chance to see that. But my issue is more related to matrix calculus than levenberg-marquardt. To put it in simpler form, what's the difference between: $\frac{\partial f(x)}{\partial x}$ and $\frac{\partial f(x)^T}{\partial x}$ Where f(x) is a R^n -> R^n vector function. – Christopher Dorian Nov 16 '10 at 15:21
This should have been a comment; anyway, you do know that differentiating a vector-valued function gives you a Jacobian. Now, the thing with both GN and LM is that your vector-valued function whose roots you're finding (or more properly, the multivariate sum-of-squares function you're minimizing) is overdetermined, that is, you have more components than variables. – J. M. Nov 16 '10 at 15:50
On the other hand, I fail to see how your question of the Jacobian of a row vector function and a column vector function relates to LM; one merely minimizes the sum-of-squares function in a manner analogous to the derivation of the normal equations, except that the Jacobian (or a modification of it, for LM) is the matrix in your overdetermined system. – J. M. Nov 16 '10 at 16:01
Thanks for the reply. I understand the concept of LM and the minimization of a sum of squared errors, I just want to strictly show the mathematical derivation. There's something wrong with my differentiation, but I don't see what I did incorrectly which leads to mismatched dimensions. – Christopher Dorian Nov 16 '10 at 23:24

I made this tutorial article about linear and nonlinear least-squares methods. It explains the problem in matrix and vector terms and I tried to make it easy to learn.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.