4
$\begingroup$

I am reading the Wikipedia article about SVM and there is something I don't understand. When they say:

These hyperplanes can be described by the equations

$$ wx - b=1 $$ and

$$wx - b=-1$$

I was wondering where does the +1 and -1 come from?

I found two papers which explain that this is an arbitrary choice:

We can write the following equations for the support hyperplanes:

$$w^T x = b + \delta $$ $$w^T x = b − \delta $$ We now note that we have over-parameterized the problem: if we scale w, b and $\delta$ by a constant factor $\alpha$, the equations for x are still satisfied.

To remove this ambiguity we will require that $\delta$ = 1, this sets the scale of the problem, i.e. if we measure distance in millimeters or meters

Source

but I don't understand what he means when he says "this sets the scale of the problem"

and

Note that if the equation $f(x) = wx + b$ defines a discriminant function

(so that the output is > $sgn(f(x))$),

then the hyperplane $cwx + cb$ defines the same discriminant function for any $c > 0$.

Thus we have the freedom to choose the scaling of $w$ so that $min_{x_i} |wx_i + b| = 1$.

Source

but I don't understand why he introduces $min_{x_i} |wx_i + b| = 1$.

My understanding is that we can do it because variables $w$, $b$ and $\delta$ are kind of linked together.

Can we change the Wikipedia definition and say

These hyperplanes can be described by the equations $$ wx - b= 2 $$ and $$wx - b=- 2$$

or is this incorrect and so we must say that :

These hyperplanes can be described by the equations $$ 2wx - 2b= 2 $$ and $$2wx - 2b=- 2$$

Could you clarify this for me?

$\endgroup$
7
+50
$\begingroup$

Very simply, if you choose any number other than 1, you can simply scale it away again. Consider $$ \mathbf{w}\cdot\mathbf{x}-b=\pm\delta $$ Now, divide both sides of the equation by $\delta$, and we get $$ \left(\frac1\delta\mathbf{w}\right)\cdot\mathbf{x}-\frac{b}{\delta}=\pm1 $$ Which means that we can define $\mathbf{\hat w}=\frac1\delta\mathbf{w}$ and $\hat b=\frac{b}{\delta}$, and we have $$ \mathbf{\hat w}\cdot\mathbf{x}-\hat b=\pm1 $$ And because the goal is to minimise $\|\mathbf{w}\|$ (in order to maximise the size of the margin, $\frac2{\|\mathbf{w}\|}$), it doesn't matter if we scale $\mathbf{w}$ by some constant such as $\delta$ first - and so, we can use $\mathbf{\hat w}$ in place of $\mathbf{w}$, and the choice of $\delta$ is irrelevant (aside from needing to be a positive real number).

Since it's irrelevant, might as well make it the simplest possible positive real number, 1.

As for why it "sets the scale" of the problem, think of it this way: changing $\delta$ would change the scaling of $\mathbf{w}$ (that is, choosing $\delta=2$ would make $\mathbf{w}$ twice as big, for example). And so, just as changing $\delta$ changes the scale, so too does setting $\delta$ set the scale - it keeps it fixed, rather than having it vary from instance to instance.

$\endgroup$
  • $\begingroup$ Thanks Glen....! $\endgroup$ – Pugl Mar 29 '18 at 13:19
  • $\begingroup$ Perfect! You cleared my confusions. +1 $\endgroup$ – Kaushal28 Apr 1 '19 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.