I am trying to do simplified version of Levenberg-Marquardt alg. for NN with one hidden layer. I found this article: Efficient algorithm for training neural networks with one hidden layer

I am not very math-skilled, so I have problem with calculating Jacobian partial derivates for matrix (7) in article (that is simplified Jacobian matrix). Does anonyone know this algorithm, or its implementation ? Classic (full) Levenberg-Marquardt is no use for me, because of large memory footprint.

  • $\begingroup$ It is not very easy indeed. I think instead of backpropagating each component at once, you have to perform backpropagation K times for each pattern p, each time for another output component k. $\endgroup$
    – alfa
    Mar 2, 2013 at 15:14
  • $\begingroup$ Yeah.. in classic LMA, I create Jacobian matrix by backpropagating from each output for each pattern, so matrix is really "huge" for big networks (and then calculate inverse... its not possible for me). This algorithm looks quite good in memory and speed of inverse, hovewer partial derivates are calculated how ? For classic LMA, I followed this description: eng.auburn.edu/~wilambm/pap/2011/K10149_C012.pdf, where is stated how to calculate partial derivates for full Jacobian matrix.. but how to change that to this modified version... ? I dont know :( I need everything in easy way :) $\endgroup$ Mar 2, 2013 at 15:38
  • $\begingroup$ You have to regard each output as a network and perform backpropagation. Then, sum up the gradients over all training patterns. $\endgroup$
    – alfa
    Mar 3, 2013 at 8:57
  • $\begingroup$ So partial derivate from error is meant as sum of squared grads ? Than I calculate grads as for classis BP and just sum them, right ? $\endgroup$ Mar 3, 2013 at 14:47
  • $\begingroup$ But don't sum up over k. $\endgroup$
    – alfa
    Mar 3, 2013 at 18:33


You must log in to answer this question.

Browse other questions tagged .