A Single Constraint

Suppose I want to maximise $f(x,y)=x^2 y$ subject to constraint $g(x,y)=x^2 + y^2 = 1$.

Geometrically, we can say that from a contour plot, $f$ is maximised under the constraint at the point where the level of $f$ is tangential to $x^2+y^2=1$. This would look something like this: contour plot

We'll call the position of the tangent, where the thick black line meets the thick green line, $(x_m,y_m)$.

What can be observed is that the gradient of $f$ at this point and the gradient of $g$ at this point, are proportional. Hence, we introduce the Lagrange multiplier, $\lambda$, a constant of proportionality for this relation:

$$\nabla f(x_m,y_m) = \lambda \nabla g(x_m,y_m)$$

From this we get a system of equations and solve for the maximum.

Now that was fine, and the idea of $\nabla f$ being proportional to $\nabla g$ is easy to see, with thanks to the geometric interpretation. Where I become confused is when we start adding multiple constraints.

Multiple Constraints

Suppose I have a function $f(x,y,z)=3x-y-3z$ and I'm trying to maximise/minimise this function subject to constraints $g_1(x,y,z)=x+y-1=0$ and $g_2(x,y,z)=x^2+2z^2-1=0$.

Similar to the single constraint case, part of the process of solving this would be to say that,

$$\nabla f=\lambda_1\nabla g_1 + \lambda_2 \nabla g_2$$

And indeed, I suppose we could generalise and say that if we had some $m$ constraints, that we'd have to solve $\nabla f= \sum_{i=1}^m \lambda_i\nabla g_i$.

The Problem

However, I am struggling for a geometric interpretation of this relationship between the gradient of $f$ and the gradients of the constraints. Because I'm struggling for a geometric interpretation, I'm struggling to understand what this means at all. Why is the gradient of $f$ a combination of the gradients of the constraints?

Does anyone have perspective on this?

  • $\begingroup$ Note that with one constraint, the gradients are two dimensional vectors acting at points on contour lines. With two constraints, the gradients are three dimensional vectors acting at points on a contour surface. For three constraints one would have to 'visualize' four dimensional gradients acting at points on contour solids--a difficult visualization. $\endgroup$ – John Wayland Bales Oct 1 '19 at 20:52
  • $\begingroup$ @JohnWaylandBales Yes. I've been thinking, when we equate gradients using Lagrange multipliers, we are just creating a linear combination of vectors, right? In the one constraint case, we're just stretching/squishing one vector to make them equal. When we have multiple constraints, we're just squishing/stretching multiple vectors and taking a linear combination of them to get $\nabla f$? $\endgroup$ – Data Oct 1 '19 at 21:02
  • $\begingroup$ I suppose that is one way to think of it. $\endgroup$ – John Wayland Bales Oct 1 '19 at 21:15
  • $\begingroup$ @JohnWaylandBales Do you have a preferred way of thinking about it? $\endgroup$ – Data Oct 1 '19 at 21:33
  • $\begingroup$ No, I just try to solve the system of equations. $\endgroup$ – John Wayland Bales Oct 1 '19 at 21:35

Let $p$ be a regular point of the surface $S$ defined by the $r$ equations $$g_i(x_1,\ldots, x_n)=0\qquad(1\leq i\leq r)\ .\tag{1}$$ This means that $p$ satisfies $(1)$, and that the $r$ vectors $\nabla g_i(p)$ should be linearly independent. The surface $S$ has dimension $d=n-r$. Let $T_p$ be its tangent plane at $p$. Each tangent vector $h\in T_p$ is orthogonal to each $\nabla g_i(p)$, hence to $V:={\rm span}\bigl(\nabla g_1(p),\ldots,\nabla g_r(p)\bigr)$. By assumption this $V$ has dimension $r$, which is equal to $n-d$. It follows that $V$ is the full orthogonal complement of the $d$-dimensional $T_p$.

When the point $p$ is a conditional extremal point of $f: \>{\mathbb R}^n\to{\mathbb R}$ on $S$ then $\nabla f(p)$ has to be orthogonal to all tangent vectors $h\in T_p$, hence $\nabla f(p)$ has to be an element of $V$. This means that $$\nabla f(p)=\sum_{i=1}^r \lambda_i \nabla g_i(p)$$ for certain real numbers $\lambda_i$.


I think there's a way to have any number of constraints you can still visualize.

Suppose you have f(x,y,z) you wish to max or minimize, and you have constraints g(x,y,z) and h(x,y,z).

$\nabla f$ points in the direction of greatest increase of f. It's opposite is in the direction of greatest decrease. A change in g given a displacement $\vec{ds}$ is $dg=\nabla g \cdot \vec{ds}$. So, we have zero change in g if our displacement is orthogonal to the gradient of g.

So we get maximal change in f without changing g if our displacement is parallel to the gradient of f, and we remove the component parallel to the gradient of g.

So $\nabla f- \frac{\nabla f \cdot \nabla g}{\nabla g \cdot \nabla g}\nabla g$ is the direction of greatest increase of f minus the component parallel to the gradient of g.

This can be generalized to multiple constraints using Graham-Schmidt Algorithm


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.