One specific explanation of intuition regarding Lagrange multipliers that I cannot wrap my head around is presented in this Medium article and in Jeff Peterson's answer here, both very popular and I think I'm missing something essential in my understanding.

The Medium article explains as this:

In reference to the following image: enter image description here

Imagine hiking from left to right on the constraint line. As we gain elevation, we walk through various level curves of f. I’ve marked two in the picture. At each level curve, imagine checking its slope — that is, the slope of a tangent line to it — and comparing that to the slope on the constraint where we’re standing.

If our slope is greater than the level curve, we can reach a higher point on the hill if we keep moving right. If our slope is less than the level curve — say, toward the right where our constraint line is declining — we need to move backward to the left to reach a higher point.

When we reach a point where the slope of the constraint line just equals the slope of the level curve, we’ve moved as high as we can. That is, we’ve reached our constrained maximum. Any movement from that point will take us downhill. In the figure, this point is marked with a large arrow pointing toward the peak.

At that point, the level curve f = a2 and the constraint have the same slope. That means they’re parallel and point in the same direction. But as we saw above, gradients are always perpendicular to level curves. So if these two curves are parallel, their gradients must also be parallel.

This argument doesn't quite make sense to me. First, I don't understand what it would mean to check the slope of the level curve. We have a tangent line in $\mathbb{R}^3$ that is tangent to a given level curve at some point. But then how does this relate to one slope being "greater" than another when we're comparing vectors in $\mathbb{R}^3$. I suppose you could look at the intersection and compute the tangent line of each level curve at the point of intersection, but it still doesn't allow us to compare to say one is greater/less.

Similarly, in Jeff Peterson's explanation, he explains: enter image description here

In this figure, 𝑔(𝑥,𝑦)=𝑐 is represented as linear in x and y and defines the coordinates x and y on 𝑓(𝑥,𝑦) where we must look for the maximum. The intersection of these surfaces forms the cyan (light blue) line and its maximum is exactly where the contours of 𝑓(𝑥,𝑦) are parallel to the plane.

It seems like this is a common way of thinking about Lagrange multipliers intuitively but similarly in this post I'm so confused on the notion of how there are contours parallel to the plane when it seems like the blue plane would lie perpendicular to the contour lines.

Am I thinking about all this wrong? I have somewhat of an intuition of Lagrange multipliers from Wikipedia but I'd like to understand what I'm missing here in these explanations.

  • $\begingroup$ Is the picture from the Medium article showing the gradient of $f$ as a vector in $\mathbb R^3$ rather than in $\mathbb R^2$? If so, that is not correct. $\endgroup$
    – littleO
    May 1 '19 at 13:22
  • $\begingroup$ @littleO I'm trying to figure that out too. The gradient should be a vector in $\mathbb{R}^2$ but i was thinking that maybe I was missing something from the picture and the author was trying to point out an idea that I was missing. $\endgroup$
    – rb612
    May 1 '19 at 17:59
  • $\begingroup$ I don't think you're missing anything, the picture is wrong because it shows the gradient of $f$ as a vector in $\mathbb R^3$ rather than $\mathbb R^2$. I think the picture of the plane that is supposed to represent the constraint $g(x_1,x_2) = c$ does not make sense either. The level set $g(x_1, x_2) = c$ is a curve in $\mathbb R^2$. $\endgroup$
    – littleO
    May 1 '19 at 19:12
  • $\begingroup$ @littleO thanks for the clarification. How about the idea of the slope being greater or this idea of the contours being parallel (to what seems like the constraint plane)? Neither of those make sense to me. $\endgroup$
    – rb612
    May 1 '19 at 19:13
  • $\begingroup$ I would just skip that Medium article; I think more clear explanations can be found. Figure 1 in the Wikipedia article is good. $\endgroup$
    – littleO
    May 1 '19 at 19:16

Some ideas.

Assuming $f, g$ are nice functions and that $g(x_1,x_2)$ has inverse inside a convenient ball $|x-\bar x| < \rho$ we have inside this ball $g(x_1,x_2) = 0 \Leftrightarrow x_1 = \phi(x_2)$

So the problems

$$ \min(\max)_{x_1,x_2} f(x_1,x_2)\ \ \mbox{s. t.}\ \ g(x_1,x_2) = 0 $$


$$ \min(\max)_{x_1,x_2} f(x_1,x_2)\ \ \mbox{s. t.}\ \ x_1-\phi(x_2) = 0 $$


$$ \min(\max)_{x_2} f(\phi(x_2),x_2) $$

are equivalent.

For the last we have the stationary points as

$$ \phi'f_{x_1}+f_{x_2} = 0 $$

and the stationary points for $f(x_1,x_2) + \lambda(x_1-\phi(x_2))$ are obtained as a solution for

$$ \nabla f+\lambda\nabla(x_1-\phi(x_2)) = \left\{\begin{array}{rcl}f_{x_1}+\lambda = 0\\ f_{x_2}-\lambda \phi'(x_2) = 0\end{array}\right. $$

After the $\lambda$ elimination we have again

$$ \phi'f_{x_1}+f_{x_2} = 0 $$

so the set of solutions for

$$ \min(\max) f(x_1,x_2)\ \ \mbox{s. t.}\ \ g(x_1,x_2) = 0 $$

and for

$$ \nabla f(x) + \lambda \nabla g(x) = 0 $$

have common points. Those points require further qualification.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.