3
$\begingroup$

I am learning support vector regression but cannot fully understand the rational of the slack variable tricks in its formulation. The original optimization problem for SVR is as follows:

$\mathrm{min}\left\{C\sum_{i=1}^NL_\epsilon\left(y_i,w_0+\mathbf{w}^T\mathbf{x}_i\right)+\frac{1}{2}||\mathbf{w}||^2\right\}$

where $L_\epsilon\left(y_i,w_0+\mathbf{w}^T\mathbf{x}_i\right)=\mathrm{max}\left\{0,\big|y_i-\left(w_0+\mathbf{w}^T\mathbf{x}_i\right)\big|-\epsilon\right\}$ is the $\epsilon$-insensitive error function. Then all the papers and textbooks I read say to introduce two slack variables $\xi_i^+$ and $\xi_i^-$ such that the above problem transforms to:

$\mathrm{min}\left\{C\sum_{i=1}^N\left(\xi_i^++\xi_i^-\right)+\frac{1}{2}||\mathbf{w}||^2\right\}$ s.t. $\xi_i^+\geq0,\xi_i^-\geq0,\xi_i^++\epsilon\geq y_i-\left(w_0+\mathbf{w}^T\mathbf{x}_i\right)\geq-\xi_i^--\epsilon$

However, I just don't see the necessity to introduce two slack variables instead of one. In fact, if we simply let $\xi_i=L_\epsilon\left(y_i,w_0+\mathbf{w}^T\mathbf{x}_i\right)$, the original problem can be written as:

$\mathrm{min}\left\{C\sum_{i=1}^N\xi_i+\frac{1}{2}||\mathbf{w}||^2\right\}$ s.t. $\xi_i=\mathrm{max}\left\{0,\big|y_i-\left(w_0+\mathbf{w}^T\mathbf{x}_i\right)\big|-\epsilon\right\}$

The above problem is equivalent to

$\mathrm{min}\left\{C\sum_{i=1}^N\xi_i+\frac{1}{2}||\mathbf{w}||^2\right\}$ s.t. $\xi_i\geq0,\xi_i+\epsilon\geq y_i-\left(w_0+\mathbf{w}^T\mathbf{x}_i\right)\geq-\xi_i-\epsilon$

That is, we can just use one slack variable so as to write this in a standard quadratic programming form. Am I wrong? If not, why go all the way round to make two slack variables? Does it render any computational vantage or just for aid of interpretation of the concept?

$\endgroup$
3
  • $\begingroup$ I know you would have no reason to expect that this is a duplicate, but does this help? math.stackexchange.com/questions/1021586/… $\endgroup$ Mar 8 '15 at 17:30
  • $\begingroup$ And I am almost positive the two slack question has been answered before, but for the life of me I cannot find it. $\endgroup$ Mar 8 '15 at 19:20
  • $\begingroup$ Thank you for your answer. I actually am aware of the post you share. Indeed I notice that in the case of classification, only one slack variable is used instead of two. So this is the reason why I wonder there shouldn't be one slack variable in the case of regression as well. $\endgroup$
    – maycui
    Mar 9 '15 at 1:34
4
$\begingroup$

I ran into the same question studying SVR, and even if this post is 2 years old maybe it can help others so here is an answer.

The slack variables in SVR are defined as such:

-> ξi+ is 0 if the training point is below the upper bound and positive if above

-> ξi- is 0 if the training point is above the lower bound and positive below

So you can see that the definitions are contradictory. If we used only 1 slack variable, say ξi+, if it was far below the lower bound, the value would still be 0. Look at the image below to convince yourself.

illustration of ξ+ and ξ-

$\endgroup$
2
$\begingroup$

Axelle's answer explains how the two slack variables are different. We could replace the two slack variables by 1 by using the absolute value of the difference between the prediction and the target variable. This would make for a non-differentiable constraint function(which could be bothersome if one needs to derive the dual formulation or while deriving the KKT conditions) and this is why two different slack variables are introduced in the regression problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.