Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I was studying Lagrange multipliers. However, I have some confusion. Let's say I have a function $f(x,y)$ to be minimized and I have some constraints $g(x,y) = 0$.

If I minimize the function $$ L(x,y,\lambda) = f(x,y) + \lambda g(x,y) \>, $$ then how does it include the constraint $g(x,y) = 0$. The book says that if I minimize $L$ with respect to $\lambda$ then it will be equivalent to minimize the function $f(x,y)$ with the constraint $g(x,y)$.

I need some clarifications.

Further it is said that

gradient(f)+ lambda * gradient(g) = 0 ............(1)

leads to

L(x,y,lambda) = f(x,y) + lambda * g(x,y)...........(2)

I didn't get this portion how come equation 1 led to equation 2?

Also I am a bit confused when it comes to inequality constraints like

g(x,y) >= 0

It is being said that f(x,y) will be maximum if its gradient is oriented away from the region g(x,y) > 0 and therefore

gradient(f(x,y)) = - lambda * gradient(g(x,y))

I just didn't get this.

share|improve this question

migrated from stats.stackexchange.com Jun 24 '12 at 8:54

This question came from our site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

    
Please remark that the method of Lagrange multipliers simply gives a condition to find critical points of $f$ constrained to $g^{-1}(0)$. Free critical points of $L$ needn't be minima. –  Siminore Jun 24 '12 at 10:53

2 Answers 2

Setting the partial derivative of L with respect to lambda f to 0 forces g(x,y)=0. Requiring partial of L with respect x and y to 0 will lead to a local extreme point subject to g(x,y) = 0. Because of the form of L this could be a minimum.

share|improve this answer

I will show you an example if the formulation of the minimization problem was of a single variable as: $f(x)+\lambda g(x)$

Now to find the lambda's first solve a closed form for x by setting the gradient w.r.t x as zero. You will have a closed form for x, containing the lambda's.

Now consider this to be $x^{*}=c(\lambda)$. Now substitute the closed form for $x^{*}$ in the constraint as, $g(x^{*})=0$ and solve for $\lambda$ which would give you a $\lambda$ that can enforce your constraint at the optimal value of $x^*$.

In a statistical modeling scenario though, the $\lambda's$ are estimated by cross-validation if f(.) and g(.) were loss functions required to be optimized over random variables. But I am not sure about the domain of your work.

share|improve this answer
    
This answer seems to make quite strong (but, unstated) assumptions about $f$ and $g$ –  cardinal Jun 24 '12 at 3:02
    
@cardinal, Are you indicating about the statement over the cross-validation and over existence of closed-forms, and over existence of a unique minimum? Also at user31820, What is the range of the function $g(x,y)$? This was a start , and the answer can be modified based on further inputs and discussion. –  user23600 Jun 24 '12 at 3:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.