1
$\begingroup$

I am trying to understand how the Lagrange multipliers method work for constrained optimization. Let's assume that we have a function $f(x)$ which is $f:\mathbb{R}^D \mapsto \mathbb{R}$. Now we have another $D$ dimensional fuction $g(x)$. We want to optimize $f(x)$ such that $x$ is subject to $g(x)=0$.

Now, I am trying to find my own way from here and to reach the Lagrange multiplier definition. For each $x$ which is on the constraint surface $g(x)=0$ we know that $\nabla g(x)$ is always the perpendicular to the constraint surface. At any point $x$ on the constraint surface, in order to move another point on the surface, we need to move in any direction $\vec{u}$ such that $\nabla g(x) . \vec{u}=0$. So, all these instantaneous movement vectors all lie on the tangent plane to the constraint surface at point $x$.

Then, for a point $x'$ on the constraint surface to be maximimum or minimum with respect to $f(x)$, I intuitively think so: In a close proximity of $x'$, for all $x$ which belongs to the constraint surface, it is either $f(x') > f(x)$ or $f(x') < f(x)$.

Then, unable to find out a rigorous explanation, I came up with the following idea: If $x'$ is a local extreme point within the constraint surface, then all of the directional derivatives of $f$ at $x'$, on the directions perpendicular to $\nabla g(x')$, must be equal to zero. This idea came up from the following analogy: If we had an extremum in the unconstrained case, all directional derivatives should gave zero. But since we are constrained on the surface of $g(x)=0$, only the derivatives on the directions which we can move to, should gave zero. And these directions are all perpendicular to $\nabla g(x')$. So, it should be $\nabla f(x') . \vec{u}=0$ for each $\nabla g(x').\vec{u}=0$. This means that the gradients of $f$ and $g$ must be parallel or anti-parallel at the location of a constrained extremum, then the regular definition of Lagrange multipliers follow.

Now I wonder whether this explanation is true? If so, how we can show that, at a local constrained extreme point $x'$ that all directional derivatives of $f$ on the directions $\vec{u}$, which are perpendicular to $\nabla g(x')$, must be equal to $0$? I simply could not find a rigorous explanation for that.

$\endgroup$
1
0
$\begingroup$

One approach is to make a change of variable: in a neighborhood of $x$, you should be able to replace one of the $D$ coordinate functions with $g(x)$ instead, so that the surface $g(x) = 0$ is just a neighborhood of $\mathbb{R}^{D-1}$. The general theorem that says you can do things like this is the implicit function theorem.


Although without thinking it through, it's probably more direct to show that if one of the directional derivatives is nonzero, that you can find points $y$ near $x$ that are roughly in that direction (so you know their value is larger or smaller than $f(x)$) and still satisfy $g(y) = 0$.

$\endgroup$
3
  • $\begingroup$ Yes; I was thinking along the same direction as your second paragraph. This implies that all directional derivatives parallel to the tangent plane of the constraint surface must be zero. But how to show this rigorously? $\endgroup$ – Ufuk Can Bicici Nov 12 '14 at 10:50
  • $\begingroup$ @UfukCanBiçici, the IFT says that locally there is an $h$ s.t. g(h(z))=0. $\endgroup$ – Martín-Blas Pérez Pinilla Nov 12 '14 at 10:59
  • $\begingroup$ I have heard about Implicit function theorem, but how should it be used here, I do not see completely. $\endgroup$ – Ufuk Can Bicici Nov 12 '14 at 11:05
0
$\begingroup$

The geometric interpretation is clear with this easy example: maximum of $f(x,y)=x+y$ with $x^2+y^2=1$. Draw $x^2+y^2=1$ and the level curves of $f$, $x+y=c$. We want maximum/minimum possible $c$ between the level curves that have nonempty intersection with $x^2+y^2=1$. That is...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.