2
$\begingroup$

Consider a nonlinear optimization problem of the form \begin{align} \min_{x}&\quad f(x)\\ \nonumber \text{subject to } \quad&h_i(x) = 0,\,i=1,\ldots,I\\ \nonumber \quad&g_j(x) \le 0,\,j=1,\ldots,J\\ \nonumber \quad&x\in X\subseteq\mathbb{R}^n \end{align}

I understand the intuitive meaning of the dual variable associated with an inequality constraint, $h_i(x)\le0$ -- it can be interpreted as the change in the objective value when we relax the constraint by a small amount (say change $0$ to some $\alpha>0$).

On the other hand, the dual variables for equality constraints are, in general, unrestricted in sign. Can someone provide some intuition for what the dual variables of an equality constraint mean?

$\endgroup$
2
$\begingroup$

Here's the way I think of it: in unconstrained optimization, you can think of $-\nabla f$ as a "force" that pulls any point in $\mathbb{R}^n$ towards the local minimum. You are at the minimum when $\nabla f =0$.

Now suppose you have some constraints (equality or inequality). The point feels a force in the $-\nabla f$ direction, but is constrained to move tangentially to the level sets of any active constraints at the current point. The point is at a minimum when $\nabla f$ is in the span of the $\nabla h$ and the $\nabla g$ for the active inequality constraints.

The $\lambda \nabla h$ can be thought of as a set of forces just strong enough to exactly counteract the components of $-\nabla f$ that would push the point into the inadmissible region. If the constraint gradients are orthogonal then $\lambda \nabla h$ is the infinitesimal change in the value of the constraint $h(x)$ due to infinitesimally moving $x$ in the $-\nabla f$ direction. The sign of $\lambda$ tells you whether this motion would take the point into the region $h<0$ or $h>0$, i.e. whether the constraint is "pushing" or "pulling" the point to keep it on $h=0$. (Inequality constraints can only push which is why their Lagrange multipliers are always positive.) In general this interpretation is not quite accurate, since the $\nabla h$ are not usually orthogonal to each other: the force from constraint $h_1$ can help or hinder keeping $x$ out of $h_2$'s inadmissible region.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.