# How to derive the Levenberg–Marquardt algorithm with matrix calculus

According to the wikipedia article: http://en.wikipedia.org/wiki/Levenberg_Marquardt

--

$S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2$

Taking the derivative with respect to δ and setting the result to zero gives:

$(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)])$

--

My attempt to derive the equation:

$\|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2 = (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

using product rule:

$\frac{\partial \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2}{\partial \boldsymbol\delta} = (-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta) + (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

The dimensions of the left and right side don't match. I believe there might be something wrong with my differentiation. There seems to be a transpose missing, but I'm not sure what would cause a transpose in the differentiation operation.

-
You did have a look at that comment I left in your MO question, yes? – J. M. Nov 16 '10 at 13:29
Yes, I had a chance to see that. But my issue is more related to matrix calculus than levenberg-marquardt. To put it in simpler form, what's the difference between: $\frac{\partial f(x)}{\partial x}$ and $\frac{\partial f(x)^T}{\partial x}$ Where f(x) is a R^n -> R^n vector function. – Christopher Dorian Nov 16 '10 at 15:21
This should have been a comment; anyway, you do know that differentiating a vector-valued function gives you a Jacobian. Now, the thing with both GN and LM is that your vector-valued function whose roots you're finding (or more properly, the multivariate sum-of-squares function you're minimizing) is overdetermined, that is, you have more components than variables. – J. M. Nov 16 '10 at 15:50
On the other hand, I fail to see how your question of the Jacobian of a row vector function and a column vector function relates to LM; one merely minimizes the sum-of-squares function in a manner analogous to the derivation of the normal equations, except that the Jacobian (or a modification of it, for LM) is the matrix in your overdetermined system. – J. M. Nov 16 '10 at 16:01
Thanks for the reply. I understand the concept of LM and the minimization of a sum of squared errors, I just want to strictly show the mathematical derivation. There's something wrong with my differentiation, but I don't see what I did incorrectly which leads to mismatched dimensions. – Christopher Dorian Nov 16 '10 at 23:24