# Why are most Lagrange multipliers zero in the SVM solution?

I read everywhere that a non-zero Lagrange multiplier $\lambda_i$ signifies that the corresponding point $x_i$ is a support vector, but I can't see how a support vector and a non-support vector have a different value for the Lagrange multiplier.

Can you please explain how the process of optimizing the Lagrangian leads to some Lagrange multipliers being zero and some non-zero?

• I think you'll find this helpful: engr.mun.ca/~baxter/Publications/LagrangeForSVMs.pdf May 27, 2016 at 21:52
• @AlexR. In 4.1 (Example 4): First it is stated that $x^2-1 \geq 0$ but later on I see that, after deriving w.r.t. $\lambda$, we get $x^2-1=0$. How is this possible? Can't this value be greater than zero? Jun 1, 2016 at 7:22
• I think you can find this answer helpful. stats.stackexchange.com/questions/54976/… Jul 3, 2020 at 20:16

$$L(x)=f(x)-\sum_k \lambda_k c_k(x),$$
where your constraint satisfies $c_k(x)\geq 0$ and $\lambda_k\geq 0$. The optimum is achieved when the gradient of the above lagrangian is equal to 0 and $\lambda_i\geq 0$ and $\lambda_i c_i(x)=0$ for all $i$. Specifically, when $\lambda_i\neq 0$, the constraint is said to active, whereas if $\lambda_i=0$, then you can freely move out of the constraint region while preserving the optimum. This is why we demand $\lambda_i>0$.
• I see that it can move freely around when $c_i(x)=0$, because then the constraint $\lambda_i c_i(x)=0$ is already satisfied. But why would it become greater than zero? Is it because then the Lagrangian is minimized? May 28, 2016 at 8:51