If your system is smooth, I would try to use one of the following solvers: KNITRO, SNOPT, ALGENCAN,IPOPT. The first two are commercial (but they are free for academics with, say, up to 300 variables), the last two are free and open source. They are all optimization solvers, so you could, for example, look for min-norm solution subject to the constraints given by your system.
I would experiment with these solvers from inside a modeling environment like AMPL or GAMS. The benefit is that you write down your model once, and to change solvers you only change one line of code (maybe two, if you start passing options to the solver). I know AMPL very well, and it has a very clear, intuitive language. They have excellent documentation in the form of the AMPL book (now available for free online on the AMPL website). It also implements very good automatic differentiation algorithms, so you don't have to provide Jacobians or Hessians, nor do you have to use finite-difference approximations. I think GAMS I believe both AMPL and GAMS have trial versions, and they are free for students with small problems (<300 variables). Another option is AIMMS, but they only support Windows and Linux (I have a Mac).
As for references, the following books have helped me in the past:
- "Numerical Optimization" by Nocedal and Wright.
- "Practical Optimization" by Gill, Murray and Wright.
- "Practical Methods of Optimization" by Fletcher.
- "Iterative Methods for Linear and Nonlinear Equations" by Kelley available for free on the site of SIAM, the publisher. There is also a similar one about optimization.
- "Solving Nonlinear Equations with Newton's Method", also by Kelley, with Matlab code.
Good luck. And next time, let us know if the problem is smooth, polynomial, etc. Not everyone will be familiar with the field/application where the problem/equations are coming from (like me, coming from an Economics/Math background), but they may still help you anyway.
Edit: the following is a very good reference on Nonlinear Programming methods and software that might be useful. I know you are solving system of equations, but you can always an optimization solver to solve a feasibility problem (constant objective, thus solving only the constraints given by your nonlinear system) or, as you have degrees of freedom, impose some objective (like norm minimization).
Two other useful websites (again, in optimization, but again, it may be useful, for reasons already stated):
- Benchmarks of different solvers by Hans Mittemman
- The NEOS Optimization Guide. An overview of algorithms, solvers, etc.
- The NEOS Server. A place where you can freely submit your optimization problems to be solved by various different solvers. Supports problems modeled mostly in AMPL or GAMS, but a few solvers also support other languages.
The NEOS server should allow you to experiment with different algorithms/software to solve your problem. The first three items above should give you some guidance in which methods to choose. Hope it helps.