Understanding Langrange Multiplier I am trying to understand Lagrange Multiplier. I think I have grasped the theory and can follow less difficult examples but I feel I am still missing full understanding. I think to optimize a function $f(\mathbf{x})$ subject to constraint $g(\mathbf{x})=C$, I can build a new function, Lagrangian, as follows:
$$L(\mathbf{x},\lambda)=f(\mathbf{x})-\lambda \left(g(\mathbf{x})-C\right)$$
If I take gradient of the Lagrangian, I'll get a vector function of derivatives ($D+1$) while $D$ is dimension of $\mathbf{x}$: 
$$
\nabla L(\mathbf{x},\lambda)=
\begin{bmatrix}
\frac{\partial L(\mathbf{x},\lambda)}{\partial x_1} \\
\vdots\\
\frac{\partial L(\mathbf{x},\lambda)}{\partial x_d} \\
\frac{\partial L(\mathbf{x},\lambda)}{\partial \lambda} \\
\end{bmatrix}
=
\begin{bmatrix}
\frac{\partial f(\mathbf{x})}{\partial x_1}-\lambda \left[ \frac{\partial }{\partial x_1} \left(g(\mathbf{x})-C \right) \right]\\
\vdots\\
\frac{\partial f(\mathbf{x})}{\partial x_d}-\lambda \left[ \frac{\partial }{\partial x_d} \left(g(\mathbf{x})-C \right) \right]\\
-\left(g(\mathbf{x})-C\right) \\
\end{bmatrix}
=0
$$
Problem.
I believe gradients of Lagrangian are equal to 0 because we want to find a point(s) $\mathbf{x}_{\,0}$ for which gradients are proportional, having the same direction. I followed a couple of theory explanations and cannot figure out why we want gradients to have the same direction, why do we look for a "point" where gradients are proportional. If I add multiple constraints, each constraint add its own gradient, how do we find "the point of the same gradient" when we have multiple gradients from different constraints?
Example.
Also, I tried a very trivial example and I failed to understand its output. I guess it is because I don't fully understand how to apply constraints using Lagrange Multipliers. For instance:
$f(x)=(x-1)(x-5)=x^2-6x+5 \\
g(x)=x-3 \\
L(x)=x^2-6x+5 - \lambda (x-3) \\
$
This is a parabola and I was looking for max value. If I don't apply constraint, max of this function is in infinity! I tried to apply a line $g(x)=x-3$ as a constraint. I build the Lagrangian, calculated its derivatives and have everything equal to 0, I have following result:  
$\begin{cases}
\frac{\mathrm dL(x)}{\mathrm d x} = 2x-6-\lambda = 0\\
\frac{\mathrm dL(x)}{\mathrm d \lambda} = x-3 = 0\\
\end{cases}$
This is awkward because the first and the second equations are the same if I don't have lambda. Also, substituting second to the first cause $\lambda=0$. When I draw parabola and the line, I have two points of intersection. I don't know why the Lagrange Multiplier doesn't work here, giving me a point $x_0 \approx 5.5$.
Thanks 
 A: The Lagrangian function is a purely formal object not having any intuitive interpretation. Setting $\nabla L=0$ together with the constraint furnishes the points ${\bf x}\in S$ (the manifold defined by the constraint) where
$$\nabla f({\bf x})=\lambda\nabla g({\bf x})\tag{1}$$
for some factor $\lambda$. These points are the conditionally stationary points of $f$ on $S$.
Now the condition $(1)$ has an intuitive geometric meaning: Consider a point ${\bf p}\in S$. Since $S$ is a level surface of the constraint function $g$ the gradient $\nabla g({\bf p})$ (assumed to be $\ne{\bf 0}$) is orthogonal to $S$ at ${\bf p}$, or more precisely: is the normal of the tangent hyperplane $S_{\bf p}$. 
On the other hand, if ${\bf p}$ is a conditionally stationary point of $f$, then  the directional derivative $$\lim_{t\to0+}{f({\bf p}+t{\bf A})-f({\bf p})\over t}=\nabla f({\bf p})\cdot{\bf A}$$ of $f$ at ${\bf p}$ is $=0$ in all allowed directions, i.e., in all directions ${\bf A}\in S_{\bf p}$. This  means that  $\nabla f({\bf p})\perp S_{\bf p}$, hence $\nabla f({\bf p})$ is parallel to $\nabla g({\bf p})$, and this is what $(1)$ is saying.
A: In your example you optimize the function $f(x)=(x-1)(x-5)$ subject to the constraint $x=3$ !!!
Hence you are looking for $\max \{f(x):x=3\}= \max \{-4\}=-4$
For this problem you really do not need Lagrange Multipliers.
A: Let me try and help you to grasp the whole scenario.
If you have to find the local max/min/inflection points of a $f({\bf x})$, it
means that you are looking for the points where $df=0$ along any direction
around the point $\bf x$.
Since
$$
0 = df = {{\partial f} \over {\partial x_1 }}dx_1  +  \cdots  + {{\partial f} \over {\partial x_n }}dx_n  = \Delta _{\,{\bf x}} f({\bf x}) \cdot d{\bf x}
$$
and the vector $d{\bf x}$ has "all the degrees of freedom", then that implies that $\Delta _{\,{\bf x}} f({\bf x})$ shall be null.
Now consider the constrain given by that $\bf x$ shall stay on
$$
h({\bf x}) = g({\bf x}) - C = 0
$$
That means that now we won't take whichever $d({\bf x})$ to set the nullity of $df$, but only 
those for which
$$
{{\partial h} \over {\partial x_1 }}dx_1  +  \cdots  + {{\partial h} \over {\partial x_n }}dx_n  = 0
$$
i.e.
$$
d{\bf x}^ *  :\Delta _{\,{\bf x}} h({\bf x}) \cdot d{\bf x}^ *   = 0
$$
Therefore
$$
\eqalign{
  & \left\{ \matrix{
  \Delta _{\,{\bf x}} h({\bf x}) \cdot d{\bf x}^ *   = 0 \hfill \cr 
  \Delta _{\,{\bf x}} f({\bf x}) \cdot d{\bf x}^ *   = 0 \hfill \cr}  \right.\quad  \Rightarrow   \cr 
  &  \Rightarrow \quad \Delta _{\,{\bf x}} f({\bf x})//\Delta _{\,{\bf x}} h({\bf x})\quad  \Rightarrow \quad \Delta _{\,{\bf x}} f({\bf x}) = \lambda \,\Delta _{\,{\bf x}} h({\bf x})\quad  \Rightarrow \quad   \cr 
  &  \Rightarrow \quad \Delta _{\,{\bf x}} \left( {f({\bf x}) - \lambda h({\bf x})} \right) = \,0\quad  \Rightarrow \quad \left( {f({\bf x}) - \lambda h({\bf x})} \right) = c \cr} 
$$
so, the Deltas shall be parallel, or $\left( {f({\bf x}) - \lambda h({\bf x})} \right) = c $ where the constant $c$ does not depend either on $\bf x$
either on $\lambda$.
From here I think you can proceed by yourself.
