Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am interested in the linear least squares problem: $$\min_x \|Ax-b\|^2$$

Without constraint, the problem can be directly solved. With an additional linear equality constraint, the problem can be directly solved too, thanks to a Lagrange multiplier.

However, I could not find a direct way to solve this problem with a linear inequality constraint. The problem belongs to quadratic programming, and the methods mentioned on Wikipedia involve an iterative procedure. I also know about Karush–Kuhn–Tucker conditions, but I cannot deal with them in this particular problem since the primal and dual feasability conditions, and the complementary slackness conditions, are numerous and not helpful in an abstract setting.

Let us assume the linear inequality constraint is indeed a simple enforcement of non-negativity: $$x\geq0$$

Is there a direct method which could be directly applied to this simpler case? The only hope I could find so far lies in the method explained in a technical report by Gene H. Golub and Michael A. Saunders, released in May 1969, called Linear Least Squares and Quadratic Programming, and which was linked in another question.

share|cite|improve this question
Lagrange multipliers account for inequality constraints as well. – Jacob Oct 25 '12 at 21:54
Then we talk about Karush–Kuhn–Tucker conditions. – Wok Oct 25 '12 at 21:57
up vote 2 down vote accepted

You are asking about the nonnegative least squares problem. Matlab has a function to solve this type of problem. Here is a paper about an algorithm to solve nonnegative least squares problems:

Arthur Szlam, Zhaohui Guo and Stanley Osher, A Split Bregman Method for Non-Negative Sparsity Penalized Least Squares with Applications to Hyperspectral Demixing, February 2010

I also think the proximal gradient method can be used to solve it pretty efficiently.

share|cite|improve this answer
Thanks for the answer. I will have a look at the article. However, keep in mind I would be interested in a direct formula rather than an iterative procedure. If this is not possible, then be it, I will try the iterative procedure. – Wok Oct 25 '12 at 21:42
I'm pretty sure the existence of this Matlab function (and the iterative algorithm given in the paper, although it solves a slightly harder problem) mean that no direct formula exists. Pretty sure you have to use an iterative procedure. – littleO Oct 25 '12 at 22:13

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.