# Handling singular matrices in gradient-descent optimization.

Right now I am coding up optimization for a 70 dimension nonlinear optimization, where the analytical gradient is unavailable.

I have some non-linear constraints that maps the structural parameters into reduced-form matrices in a non-trivial way. Sometimes one of the reduced-form matrices is close to singular, and I need to take the inverse of the matrix.

To avoid the issue, I currently check the conditioning number of the matrix in question, and if it is sufficiently large I create a discontinuity in the objective function.

I've learned that gradient-descent algorithms don't play well with discontinuities, but I am not sure what the best way to code this constraint with my solver.

BTW I am using MATLAB's FMINCON's Interior-Point solver.

• Do you really need to take the inverse of the matrix? What larger calculation are you trtying to do? – Mark L. Stone Apr 24 at 1:48
• The calculation is inv(A)*B. As I understand it, A\B will also have accuracy issues. And I've been warned against using pinv(A)*B in statistical estimation as pinv(A) may lead to weird results when the matrix in near-singular. – hipHopMetropolisHastings Apr 24 at 1:56
• A\B doesn't cure all ills, but it is better than inv(A)*b. Perhaps you can reformulate in such a way as to avoid near singularity? Have you tried FMINCON's SQP algorithm? SQP often handles difficult constraints more robustly than IPM. Also, are you using user supplied Hessian? That should also improve robustness. – Mark L. Stone Apr 24 at 2:04
• I have not tried SQP, good suggestion. Not sure about reformulation. Don't think I can use user supplied hessian as I can't derive it analytically. – hipHopMetropolisHastings Apr 24 at 2:13
• Yes. ADiMAt does source transformation on your m file containing function to produce a new function which also evaluates derivative. ADiMat sc.informatik.tu-darmstadt.de/res/sw/adimat/index.en.jsp and adimat.sc.informatik.tu-darmstadt.de can do forward over reverse for Hessian. Gradient is evaluated by reverse mode, then Hessian is evaluated as forward mode derivative of gradient. – Mark L. Stone Apr 24 at 2:42