I cannot understand how if a Lagrange multiplier is a scalar (meaning it is a constant value) that you can take a partial derivative of a Lagrangian function with respect to a constant (the Lagrange multiplier) when you need to find the gradient of a Lagrangian function.

If the derivative of a constant is zero, then how/why is it even possible to take the derivative of something with respect to a constant/Lagrange multiplier?


1 Answer 1


It's common to call something a "constant" in informal descriptions of a math problem, but it's important to always keep in mind: constant with respect to what?

Let's look at a concrete example: linear-constrained least squares,

$$\min_{\mathbf{x}}\ \sum_{i=1}^{n} (x_i - y_i)^2 \quad \mathrm{s.t.} \quad \sum_{i=1}^n x_i = b.$$ Here the vector $\mathbf{x}\in \mathbb{R}^n$ is the independent variable of the optimization problem, and $y_i$ and $b$ are constants with respect to $\mathbf{x}$.

We can reformulate this problem using the method of Lagrange multipliers: $$\underset{\mathbf{x},\lambda}{\operatorname{ext}} \ \sum_{i=1}^{n} (x_i - y_i)^2 - \lambda\left(\sum_{i=1}^n x_i - b\right).$$ In this new problem:

  • $\lambda$ does not depend on $i$; it's a single "global constant";
  • $\lambda$ is constant with respect to $\mathbf{x}$; it's an independent variable, and not a function of $\mathbf{x}$;
  • $\lambda$ is an independent variable (not a constant) of the optimization problem: you are trying to solve for $\lambda$ as part of the optimization problem, and are not given the value of $\lambda$ in advance.

Notice that there are three different notions of being "a constant" here and whether or not $\lambda$ is a constant depends on which you mean.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.