I'm learning the math behind SVMs (Support Vector Machines) from here. I understood the intuition behind Lagrange multipliers. But what I don't understand is that, why Lagrange multiplier is considered as a variable in Lagrangian?

Isn't it just a constant by which the gradient vectors of the objective function (for which the optimization is to be done) and the subject function (constraint function) vary in length?

Here is the equation of the Lagrangian:

$$L(x, y, \lambda) = f(x,y) - \lambda g(x, y)$$

where $f(x, y)$ is objective function and $g(x, y)$ is constraint function. Now applying gradient both sides:

$$\nabla L(x, y, \lambda) = \nabla f(x,y) - \lambda \nabla g(x, y)$$

Here while applying $\nabla$, we've considered $\lambda$ as constant, even if we are considering Lagrangian ($L$) as a function of $x$, $y$ and $\lambda$.

What I'm missing here? Any intuitive explanation will help. Thanks!


In Wikipedia page of this topic, $\lambda$ is considered as constant. Here is the direct quote from Wikipedia page.

The constant λ is required because although the two gradient vectors are parallel, the magnitudes of the gradient vectors are generally not equal. This constant is called the Lagrange multiplier. (In some conventions λ is preceded by a minus sign).

  • $\begingroup$ since Lagrangian is a $3$-variable function, you must take $3$ partial derivatives, so $\lambda$ is a variable (not a constant). $\endgroup$ – farruhota Mar 30 '19 at 13:00
  • $\begingroup$ @farruhota But this is what I don't understand: Why is $\lambda$ variable? And if it's variable, then why didn't we apply chain rule of derivatives while applying $\nabla$ on the right side? This is what confuses me. $\endgroup$ – Kaushal28 Mar 30 '19 at 13:03
  • $\begingroup$ @farruhota see the edits. $\endgroup$ – Kaushal28 Mar 30 '19 at 13:15
  • $\begingroup$ Keep reading, you will see $\nabla_{x,y,\lambda} L(x,y,\lambda)=\left(L_x,L_y,L_{\lambda}\right)$. Well, $L_{\lambda}=0$ is basically the constraint itself. $\endgroup$ – farruhota Mar 30 '19 at 13:24
  • $\begingroup$ Oh yes! Got it. Thanks! $\endgroup$ – Kaushal28 Mar 30 '19 at 14:08

The minimal value point $(x_0,y_0)$ must satisfies two conditions: $$g(x_0,y_0)=0\qquad \mbox{and}\qquad \nabla g(x_0,y_0)=\lambda_0 \nabla f(x_0,y_0)$$ where $\lambda_0$ is constant.

Define $L(x,y)=f(x,y)-\lambda_0 g(x,y)$, then $\nabla g(x_0,y_0)=\lambda_0 \nabla f(x_0,y_0)$ is equivalent to $$L_x(x_0,y_0)=L_y(x_0,y_0)=0$$ and in form:$$L_{\lambda_0}(x_0,y_0)=g(x_0,y_0)=0$$ so consider $\lambda$ as variable is only in order to guarantee the constraint condition $g(x_0,y_0)=0$.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the clear explanation! $\endgroup$ – Kaushal28 Mar 30 '19 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.