This question is related with machine learning. $ \\ $But I assumed linear algebra people will be able to help.

The most basic classification form

$$y(x) = w^Tx + w_0$$

$w \ $ is called weight vector.

These equations give me some trouble to understand (not good with linear algebra).

Classification linear regression :

  • $x = x_{\bot} + r \frac{w}{||w||}$
  • $r = \frac{y(x)}{||w||}$

Not sure which part of linear algebra or vector space should I look up to understand this picture.

$\frac{y(x)}{||w||}$ , $\frac{-w_0}{||w||}$ : where these guys come from and represent?

And final equation

vector equation

Can anyone help me to understand this equation simply?


Okay so the figure explains the equations $y(x) = w^T x + w_0$, for a 2D-case, namely when the vectors $w$ and $x$ are 2 dimensional. Keep in mind that vector $w$ is the normal (or directional) vector to line $w^T x$ and $w_0$ is just an intercept, i.e. it shifts the line $w^T x$ that passes thru the origin (black line) to $y(x) = w^T x + w_0$.

-The notation $x_{\perp}$ got me a bit confused, but I quite understood it by doing the following: \begin{equation} \begin{split} y(x) &= w^Tx + w_0 \\&= w^T(x_{\perp} + r\frac{w}{\Vert w \Vert}) + w_0 \\&= w^T x_{\perp} + r\Vert w \Vert + w_0 \\&= w^T x_{\perp} + \frac{y(x)}{\Vert w \Vert}\Vert w \Vert + w_0 \\&= w^T x_{\perp} + y(x) + w_0 \end{split} \end{equation} which means $w^T x_{\perp} + w_0 = 0$. Therefore, the point $x_{\perp}$ is such that $y(x_{\perp}) = 0$, therefore the red line. P.S: It got me confused because usually $x_{\perp}$ means a vector that nulls $x$, i.e. $x^T x_{\perp} = 0$, but this is not what you mean in the figure.

-First answer to: where did $\frac{y(x)}{\Vert w \Vert}$ come from ? Well, this is the distance between any point $x$ and the line $y(x_{\perp})$ (the red line). Using the following relation which gives the distance between point $x$ and line $y(x_{\perp})$, we have \begin{equation} \text{dist}(y(x_{\perp}),x) = \frac{\vert y(x) \vert }{\Vert w \Vert} \end{equation} where we can remove the $\vert \vert$ since $y(x)$ is positive in the figure above. [Reference, see https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line].

-Second answer to where did $-\frac{w_0}{\Vert w \Vert}$ come from: It is the distance between two parallel lines: The red line $y(x_{\perp})$ and the parallel black line say $z(x) = w^Tx$ since they have same directional vector $w$ and the black line passes thru the origin (so no intercept). Hence \begin{equation} \text{dist}(y(x_{\perp}),z(x)) = \frac{\vert w_0 - 0 \vert }{\Vert w \Vert} \end{equation} where the negative sign in the figure appears because it is towards the negative side. [Reference, see https://en.wikipedia.org/wiki/Distance_between_two_straight_lines].

  • $\begingroup$ thx EL, I am trying to follow your answer. can you tell me how $$w^T(r * (w/||w||) )= r||w||$$ on the middle? $\endgroup$ – Mike Kim Oct 5 '16 at 11:35
  • $\begingroup$ Yes so $w^T(r \frac{w}{\Vert w \Vert}) = r \frac{w^Tw}{\Vert w \Vert}$ since $r$ is a scalar. Also $w^Tw = \Vert w \Vert^2$. Therefore, $w^T(r \frac{w}{\Vert w \Vert}) = r \frac{w^Tw}{\Vert w \Vert} = r \frac{ \Vert w \Vert^2}{\Vert w \Vert} = r \Vert w \Vert$ $\endgroup$ – Ahmad Bazzi Oct 5 '16 at 14:45
  • $\begingroup$ oh that is clear~thx $\endgroup$ – Mike Kim Oct 6 '16 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.