The role of Tangent lines in the method of Lagrange Multipliers

Question

I have been studying the method of Lagrange multipliers for a machine learning course. From the following sources:

https://en.wikipedia.org/wiki/Lagrange_multiplier#Single_constraint

https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/lagrange-multipliers-and-constrained-optimization/v/meaning-of-lagrange-multiplier

https://www.byteofmath.com/the-math-behind-support-vector-machines/

I have been able to achieve a core understanding of the method. However this one source:

https://medium.com/@andrew.chamberlain/a-simple-explanation-of-why-lagrange-multipliers-works-253e2cdcbf74

makes it sound as though I can use comparisons between the slope of the tangent line of the constraint curve $g(x,y,z,..) = 0$ and the slope of the tangent line of the level curve $f(x,y,z..) = d$ at points where these curves intersect to determine which direction along the constraint curve I should move in order to reach a max or min of the surface $f(x,y,z,...)$ subject to the constraint curve $g(x,y,z,..) = 0$.

To quote the source:

"Imagine hiking from left to right on the constraint line. As we gain elevation, we walk through various level curves of f. ...~... .At each level curve, imagine checking its slope — that is, the slope of a tangent line to it — and comparing that to the slope on the constraint where we’re standing.

If our slope is greater than the level curve, we can reach a higher point on the hill if we keep moving right. If our slope is less than the level curve — say, toward the right where our constraint line is declining — we need to move backward to the left to reach a higher point."

Figure 1 in the Wikipedia source seems to give a counter example to this (if I am interpreting things correctly) hence my confusion. It would be a great help if this role of the tangent line in Lagrange Multipliers can be confirmed and the math behind it shown. Moreover, if this is confirmed, I would like clarification on whether the sign of the slopes matters (Which I would think that it does) or if its just the magnitude.

I realize my question is similar to and contains the same subject matter as these questions:

Lagrange multipliers tangency

Lagrange multiplier parallel slope intuition confusion?

but I'm going to post this question anyway

Thank you

Answer 1

I am by no means qualified to answer your question. However, I was struggling with the same problem and would like to share what I found:

First of, let's get some intuition with the Lagrange formula, in the bivariate case. We know that the function $g(x,y) = 0$ That is why we can rewrite $$f(x,y) = f(x,y) - \lambda 0 = f(x,y) - \lambda g(x,y)$$ This last equation is what we recognize as the Lagrange formula: $$ \mathcal{L}(x,y) = f(x,y) - \lambda g(x,y) $$ As a result, maximizing $f(x,y)$ is no different than maximizing the Lagrangian $$ D\mathcal{L}(x,y) = 0 \\ \Rightarrow Df(x,y) - \lambda Dg(x,y) = 0 \\ Df(x,y) = \lambda Dg(x,y) $$ We see that $Dg(x,y) $ is parallel to $Df(x,y)$. So, it is not the sign (or magnitude) of the derivative that matters but the direction.

Note that the Lagrange multiplier $\lambda$ is only introduced to generalize the formula: There are more solutions to $Df(A)= cDg(B)$ than $Df(a)=Dg(b)$ (seeing as $a \subset A$ and $b \subset B$). Both equations maximize $f$ under some constraint(s), so it's intuitively better to find more answers for $\max f$.

As to your question about figure 1 on wikipedia, I have deciphered the following: the arrows are orthogonal to the directional derivatives. $d_2,d_3$ are arbitrary points where for values of $(x,y)$ the constraint is met. It shows us that, only when the curves' tangents are parallel (e.g. the tangents' orthogonal complements are parallel), the function $f(x,y)$ is maximized, with $g(x,y) = 0 $.

Hope this helps.