# Understanding Lagrange multiplier

I am trying to understand Lagrange Multiplier. I think I have grasped the theory and can follow less difficult examples but I feel I am still missing full understanding. I think to optimize a function $f(\mathbf{x})$ subject to constraint $g(\mathbf{x})=C$, I can build a new function, Lagrangian, as follows:

$$L(\mathbf{x},\lambda)=f(\mathbf{x})-\lambda \left(g(\mathbf{x})-C\right)$$

If I take gradient of the Lagrangian, I'll get a vector function of derivatives ($D+1$) while $D$ is dimension of $\mathbf{x}$:

$$\nabla L(\mathbf{x},\lambda)= \begin{bmatrix} \frac{\partial L(\mathbf{x},\lambda)}{\partial x_1} \\ \vdots\\ \frac{\partial L(\mathbf{x},\lambda)}{\partial x_d} \\ \frac{\partial L(\mathbf{x},\lambda)}{\partial \lambda} \\ \end{bmatrix} = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1}-\lambda \left[ \frac{\partial }{\partial x_1} \left(g(\mathbf{x})-C \right) \right]\\ \vdots\\ \frac{\partial f(\mathbf{x})}{\partial x_d}-\lambda \left[ \frac{\partial }{\partial x_d} \left(g(\mathbf{x})-C \right) \right]\\ -\left(g(\mathbf{x})-C\right) \\ \end{bmatrix} =0$$

Problem.
I believe gradients of Lagrangian are equal to 0 because we want to find a point(s) $\mathbf{x}_{\,0}$ for which gradients are proportional, having the same direction. I followed a couple of theory explanations and cannot figure out why we want gradients to have the same direction, why do we look for a "point" where gradients are proportional. If I add multiple constraints, each constraint add its own gradient, how do we find "the point of the same gradient" when we have multiple gradients from different constraints?

Example.
Also, I tried a very trivial example and I failed to understand its output. I guess it is because I don't fully understand how to apply constraints using Lagrange Multipliers. For instance:

$f(x)=(x-1)(x-5)=x^2-6x+5 \\ g(x)=x-3 \\ L(x)=x^2-6x+5 - \lambda (x-3) \\$

This is a parabola and I was looking for max value. If I don't apply constraint, max of this function is in infinity! I tried to apply a line $g(x)=x-3$ as a constraint. I build the Lagrangian, calculated its derivatives and have everything equal to 0, I have following result:

$\begin{cases} \frac{\mathrm dL(x)}{\mathrm d x} = 2x-6-\lambda = 0\\ \frac{\mathrm dL(x)}{\mathrm d \lambda} = x-3 = 0\\ \end{cases}$

This is awkward because the first and the second equations are the same if I don't have lambda. Also, substituting second to the first cause $\lambda=0$. When I draw parabola and the line, I have two points of intersection. I don't know why the Lagrange Multiplier doesn't work here, giving me a point $x_0 \approx 5.5$.

Thanks

• The Lagrange-multiplier-methods applies if we have at least one constraint. Usually, the function has at least two variables. If we have a univariate function, we do not need the method, we just set the derivate $0$. This only gives local extrema, but the method of the Largrange-multiplier also only gives local extrema. Aug 3, 2017 at 12:34
• I do not think that it makes sense to take the derivate as a constraint, if we have no explicit constraints. Aug 3, 2017 at 12:37
• @Peter May I ask what you mean by explicit constraints? I am probably missing the point, something obvious. If I imagine the parabola and I am looking for a max value, $f(x)=x^2-6x+5$ has no maximum because it is infinite. I was wondering the line $g(x)=x-3$ would apply constraint. The point I was expecting to be $x \approx 5.5$. Aug 3, 2017 at 13:17
• Explicit constrains are equalities that must be satisfied. The mehod of the Lagrange multipliers uses them to find (local!) extrema. A constrain is not a function, so $g(x)=x-3$ cannot be a constrain. Aug 3, 2017 at 13:28
• @Peter Thanks for help. I understand now. I was confused because functions and equations are presented in the same way but your comments helped me to see the difference :). Thanks. Aug 3, 2017 at 14:05

The Lagrangian function is a purely formal object not having any intuitive interpretation. Setting $\nabla L=0$ together with the constraint furnishes the points ${\bf x}\in S$ (the manifold defined by the constraint) where $$\nabla f({\bf x})=\lambda\nabla g({\bf x})\tag{1}$$ for some factor $\lambda$. These points are the conditionally stationary points of $f$ on $S$.

Now the condition $(1)$ has an intuitive geometric meaning: Consider a point ${\bf p}\in S$. Since $S$ is a level surface of the constraint function $g$ the gradient $\nabla g({\bf p})$ (assumed to be $\ne{\bf 0}$) is orthogonal to $S$ at ${\bf p}$, or more precisely: is the normal of the tangent hyperplane $S_{\bf p}$.

On the other hand, if ${\bf p}$ is a conditionally stationary point of $f$, then the directional derivative $$\lim_{t\to0+}{f({\bf p}+t{\bf A})-f({\bf p})\over t}=\nabla f({\bf p})\cdot{\bf A}$$ of $f$ at ${\bf p}$ is $=0$ in all allowed directions, i.e., in all directions ${\bf A}\in S_{\bf p}$. This means that $\nabla f({\bf p})\perp S_{\bf p}$, hence $\nabla f({\bf p})$ is parallel to $\nabla g({\bf p})$, and this is what $(1)$ is saying.

In your example you optimize the function $f(x)=(x-1)(x-5)$ subject to the constraint $x=3$ !!!

Hence you are looking for $\max \{f(x):x=3\}= \max \{-4\}=-4$

For this problem you really do not need Lagrange Multipliers.

• What I meant by x=3 was a line not a point. I explained this in comment section to Peter. I also have problem to understand theory behind Lagrange Multiplier. I put it in the section "Problem" in my question. I cannot figure out why do we look for a point where constraint and a function have the same gradient. Aug 3, 2017 at 13:25

If you have to find the local max/min/inflection points of a $f({\bf x})$, it means that you are looking for the points where $df=0$ along any direction around the point $\bf x$. Since $$0 = df = {{\partial f} \over {\partial x_1 }}dx_1 + \cdots + {{\partial f} \over {\partial x_n }}dx_n = \Delta _{\,{\bf x}} f({\bf x}) \cdot d{\bf x}$$ and the vector $d{\bf x}$ has "all the degrees of freedom", then that implies that $\Delta _{\,{\bf x}} f({\bf x})$ shall be null.
Now consider the constrain given by that $\bf x$ shall stay on $$h({\bf x}) = g({\bf x}) - C = 0$$ That means that now we won't take whichever $d({\bf x})$ to set the nullity of $df$, but only those for which $${{\partial h} \over {\partial x_1 }}dx_1 + \cdots + {{\partial h} \over {\partial x_n }}dx_n = 0$$ i.e. $$d{\bf x}^ * :\Delta _{\,{\bf x}} h({\bf x}) \cdot d{\bf x}^ * = 0$$
Therefore \eqalign{ & \left\{ \matrix{ \Delta _{\,{\bf x}} h({\bf x}) \cdot d{\bf x}^ * = 0 \hfill \cr \Delta _{\,{\bf x}} f({\bf x}) \cdot d{\bf x}^ * = 0 \hfill \cr} \right.\quad \Rightarrow \cr & \Rightarrow \quad \Delta _{\,{\bf x}} f({\bf x})//\Delta _{\,{\bf x}} h({\bf x})\quad \Rightarrow \quad \Delta _{\,{\bf x}} f({\bf x}) = \lambda \,\Delta _{\,{\bf x}} h({\bf x})\quad \Rightarrow \quad \cr & \Rightarrow \quad \Delta _{\,{\bf x}} \left( {f({\bf x}) - \lambda h({\bf x})} \right) = \,0\quad \Rightarrow \quad \left( {f({\bf x}) - \lambda h({\bf x})} \right) = c \cr} so, the Deltas shall be parallel, or $\left( {f({\bf x}) - \lambda h({\bf x})} \right) = c$ where the constant $c$ does not depend either on $\bf x$ either on $\lambda$.