Working on optimization problems through neural networks, I use the Levenberg-Marquardt algorithm. I have read this assertion that I do not understand :
A positive definite diagonal matrix is added to Hessian approximation that is designed to turn the matrix into a positive-definite one in order to avoid the ill-conditioning problem. Google books : gW51UqDZfB8C
The same in an other reference :
The error surface contains many saddle points, so there may be points during training where the Hessian is indefinite (i.e., has both positive and negative eigenvalues) and the condition number does not exist. ftp://ftp.sas.com/pub/neural/illcond/illcond.html
I don't get why having a positive definite matrix helps to avoid the ill-conditioning problem, because the condition number here is defined unless the matrix is singular. For instance Hilbert's matrix are both positive-definite and an exemple of ill-conditionned matrix.
But it appears to be used in LM implentations : https://gist.github.com/rbabich/3539146
Here, line 98 /* ill-conditioned ? */
the author checks the ill-conditioning in his cholesky function (line 103) : that says the matrix is ill conditioned if and only if it is not positive-definite (line 206).
So here's the problem, I'm a bit confused with all these considerations because to me the only two things that positive-definiteness adds in LM is to be sure that the chosen direction is a descent one. And that if the damping factor is huge it adds stability to the gradient descent because the matrix becomes well conditioned (lambda_max/lambda_min ~ 1)
Thank you for your attention!