I need C# code for solving constrainted non-linear least squares problems. I'm prepared to write the code myself, but I need to understand the algorithm first. Can anyone describe a constrained non-linear least squares algorithm for me, or point me to documentation elsewhere?
To elaborate, I know that the Levenberg-Marquardt algorithm is particularly well-documented, with many implementation examples available on the web. In its basic form, however, it does not support constraints. Nevertheless, some code libraries claim to offer implementations of the Levenberg-Marquardt algorithm which do support constraints (e.g. levmar). If I can find documentation for a technique that adds support for constraints to the Levenberg-Marquardt algorithm, I believe I would have what I need.
My specific requirement is that I need to support both equality and inequality constraints on linear functions of the variables. Simply enforcing bounds on the variables is insufficient.
The question was faulted for "missing context or other details", so here is some further elaboration:
I need to create a procedure for tuning the theoretical models used for the operational planning of several industrial processes. The models will be tuned by adjusting model parameters to produce a good fit between the model predictions and historical observations. This is an optimization problem. Specifically, it is a constrained non-linear optimization problem with a sum-of-squares objective function that can only be evaluated numerically. I need a solver appropriate for that task.
The Levenberg-Marquardt algorithm looks like a good place to start. I have found descriptions of that algorithm on Wikipedia and in Numerical Recipes in C. However, those descriptions describe an unconstrained version. I don't know how to make the Levenberg-Marquardt algorithm (or any other non-linear least squares solver algorithm) respect constraints. The levmar code library (users.ics.forth.gr/~lourakis/levmar) claims to offer an implementation of the Levenberg-Marquardt algorithm that supports constraints, so it would seem possible to modify the algorithm to support constraints somehow. I can try to reverse-engineer the levmar code to figure it out, but I would prefer to approach this from a theoretical perspective first. Thus, I'm posting on the math forum to see if anyone can help me with the theory required to solve constrained non-linear least squares problems.
Note that I have already solved the problem using a generic non-linear solver (a solver that works with any objective function, not just sum-of-squares). I am not satisfied with the performance of that solver.