I'm bit confused between Gradient descent and convex optimization using Lagrange Multipliers. I know that we use Lagrange multipliers when we have an optimization problem with one or more constraints.

From the answer of this question, it seems that we can also use gradient descent for constrained optimization.

So what is the difference between those two approaches? Mathematically I know how both of the approaches work but I don't understand when and why one is preferred over another? For example, for optimization of SVM (Support Vector Machine) problem, we use Lagrange multipliers instead of gradient descent.

I've found one similar question here. But the answer is not much clear. Any intuitive explanation/example will help. Thanks.

  • 2
    $\begingroup$ Gradient descent does not handle constraints, but there is a variant of gradient descent called the projected gradient method that can sometimes handle constraints. In order for the projected gradient method to be useful, you must be able to project onto the feasible set efficiently. By the way, "use Lagrange multipliers" does not specify a unique optimization algorithm. There are many different optimization algorithms that compute Lagrange multipliers while also solving the primal problem. $\endgroup$
    – littleO
    Apr 9 '19 at 2:17
  • $\begingroup$ @littleO Any example algorithm which computes Lagrange multiplier for optimization? $\endgroup$
    – Kaushal28
    Apr 9 '19 at 5:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.