Lagrange Multipliers: why are the gradients of the function and the constraint not always exactly equal to each other at a maximum? I'm trying to understand the intuition behind LM better, and keep running into this conceptual wall. The method of LM relies on the fact that at the maximum, the gradient of the function to optimize is equal to some multiple of the constraint. However, whenever I try to visualize this, all I see is that at a maximum, the gradients are not only proportional by a constant, they are equal. So my question is why are the gradients not always exactly equal (in other words, why does it appear to me that the multiplier should only ever be 1)?
I mean, if you have a circle, and a line (the constraint) that just touches the circle at a certain point, then we know that their slopes are equal at that point, not just some multiple of each other.
As a quick edit: I guess I understand 2 cases of the multiplier, when it is either 1 or -1. I makes sense to me that when it's 1, the functions are parallel, and when the gradients are equal but opposite, the multiplier will equal -1. But my question still stands for multiplier values not equal to 1 or -1.
 A: Suppose that you are after the maximum of $x+y$ in the circle$$\{(x,y)\in\mathbb R^2\mid x^2+y^2=1\}.\tag1$$You think that you should be after the points $(x,y)$ such that $(1,1)=(2x,2y)$, since, if $f(x,y)=x+y$ and $g(x,y)=x^2+y^2$, you have $(1,1)=\nabla f(x,y)$ and $\nabla g(g,y)=(2x,2y)$. However, suppose that instead of $(1)$, we are dealing with$$\{(x,y)\in\mathbb R^2\mid2x^2+2y^2=2\}.\tag2$$It's the same set as $(1)$, but now, instead of $g$, you are dealing with the function $h(x,y)=2x^2+2y^2$. And so, by your approach, you should be after those $(x,y)$ such that $(1,1)=(4x,4y)$. But that doesn't make sense, since the sets $(1)$ and $(2)$ are equal.
A: $f(x,y)$ and $g(x,y)$ tangents at stationationary point, therefore their gradients share the same direction. However given $f(x,y)$ and $g(x,y)$ usually have different curventures at this point, $\triangledown f(x,y)$ and $\triangledown g(x,y)$ may not have the same amplitudes. Obviously $\triangledown f(x,y)$ crossing both sides of $f(x,y)$ at this point is the same.
