0
$\begingroup$

I have been studying the method of Lagrange multipliers for a machine learning course. From the following sources:

https://en.wikipedia.org/wiki/Lagrange_multiplier#Single_constraint

https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/lagrange-multipliers-and-constrained-optimization/v/meaning-of-lagrange-multiplier

https://www.byteofmath.com/the-math-behind-support-vector-machines/

I have been able to achieve a core understanding of the method. However this one source:

https://medium.com/@andrew.chamberlain/a-simple-explanation-of-why-lagrange-multipliers-works-253e2cdcbf74

makes it sound as though I can use comparisons between the slope of the tangent line of the constraint curve $g(x,y,z,..) = 0$ and the slope of the tangent line of the level curve $f(x,y,z..) = d$ at points where these curves intersect to determine which direction along the constraint curve I should move in order to reach a max or min of the surface $f(x,y,z,...)$ subject to the constraint curve $g(x,y,z,..) = 0$.

To quote the source:

"Imagine hiking from left to right on the constraint line. As we gain elevation, we walk through various level curves of f. ...~... .At each level curve, imagine checking its slope — that is, the slope of a tangent line to it — and comparing that to the slope on the constraint where we’re standing.

If our slope is greater than the level curve, we can reach a higher point on the hill if we keep moving right. If our slope is less than the level curve — say, toward the right where our constraint line is declining — we need to move backward to the left to reach a higher point."

Figure 1 in the Wikipedia source seems to give a counter example to this (if I am interpreting things correctly) hence my confusion. It would be a great help if this role of the tangent line in Lagrange Multipliers can be confirmed and the math behind it shown. Moreover, if this is confirmed, I would like clarification on whether the sign of the slopes matters (Which I would think that it does) or if its just the magnitude.

I realize my question is similar to and contains the same subject matter as these questions:

Lagrange multipliers tangency

Lagrange multiplier parallel slope intuition confusion?

but I'm going to post this question anyway

Thank you

$\endgroup$

1 Answer 1

1
$\begingroup$

I am by no means qualified to answer your question. However, I was struggling with the same problem and would like to share what I found:

First of, let's get some intuition with the Lagrange formula, in the bivariate case. We know that the function $g(x,y) = 0$ That is why we can rewrite $$f(x,y) = f(x,y) - \lambda 0 = f(x,y) - \lambda g(x,y)$$ This last equation is what we recognize as the Lagrange formula: $$ \mathcal{L}(x,y) = f(x,y) - \lambda g(x,y) $$ As a result, maximizing $f(x,y)$ is no different than maximizing the Lagrangian $$ D\mathcal{L}(x,y) = 0 \\ \Rightarrow Df(x,y) - \lambda Dg(x,y) = 0 \\ Df(x,y) = \lambda Dg(x,y) $$ We see that $Dg(x,y) $ is parallel to $Df(x,y)$. So, it is not the sign (or magnitude) of the derivative that matters but the direction.

Note that the Lagrange multiplier $\lambda$ is only introduced to generalize the formula: There are more solutions to $Df(A)= cDg(B)$ than $Df(a)=Dg(b)$ (seeing as $a \subset A$ and $b \subset B$). Both equations maximize $f$ under some constraint(s), so it's intuitively better to find more answers for $\max f$.

As to your question about figure 1 on wikipedia, I have deciphered the following: the arrows are orthogonal to the directional derivatives. $d_2,d_3$ are arbitrary points where for values of $(x,y)$ the constraint is met. It shows us that, only when the curves' tangents are parallel (e.g. the tangents' orthogonal complements are parallel), the function $f(x,y)$ is maximized, with $g(x,y) = 0 $.

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.