0
$\begingroup$

I read everywhere that a non-zero Lagrange multiplier $\lambda_i$ signifies that the corresponding point $x_i$ is a support vector, but I can't see how a support vector and a non-support vector have a different value for the Lagrange multiplier.

Can you please explain how the process of optimizing the Lagrangian leads to some Lagrange multipliers being zero and some non-zero?

$\endgroup$
3

1 Answer 1

1
$\begingroup$

When solving your SVM problem, you'll be optimizing a Lagrangian subject to KKT conditions. Specifically, something like:

$$L(x)=f(x)-\sum_k \lambda_k c_k(x),$$

where your constraint satisfies $c_k(x)\geq 0$ and $\lambda_k\geq 0$. The optimum is achieved when the gradient of the above lagrangian is equal to 0 and $\lambda_i\geq 0$ and $\lambda_i c_i(x)=0$ for all $i$. Specifically, when $\lambda_i\neq 0$, the constraint is said to active, whereas if $\lambda_i=0$, then you can freely move out of the constraint region while preserving the optimum. This is why we demand $\lambda_i>0$.

$\endgroup$
1
  • $\begingroup$ I see that it can move freely around when $c_i(x)=0$, because then the constraint $\lambda_i c_i(x)=0$ is already satisfied. But why would it become greater than zero? Is it because then the Lagrangian is minimized? $\endgroup$ May 28, 2016 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.