# linear regression and vector space

This question is related with machine learning. $\\$But I assumed linear algebra people will be able to help.

The most basic classification form

$$y(x) = w^Tx + w_0$$

$w \$ is called weight vector.

These equations give me some trouble to understand (not good with linear algebra).

Classification linear regression :

• $x = x_{\bot} + r \frac{w}{||w||}$
• $r = \frac{y(x)}{||w||}$

Not sure which part of linear algebra or vector space should I look up to understand this picture.

$\frac{y(x)}{||w||}$ , $\frac{-w_0}{||w||}$ : where these guys come from and represent?

And final equation

Can anyone help me to understand this equation simply?

Okay so the figure explains the equations $y(x) = w^T x + w_0$, for a 2D-case, namely when the vectors $w$ and $x$ are 2 dimensional. Keep in mind that vector $w$ is the normal (or directional) vector to line $w^T x$ and $w_0$ is just an intercept, i.e. it shifts the line $w^T x$ that passes thru the origin (black line) to $y(x) = w^T x + w_0$.
-The notation $x_{\perp}$ got me a bit confused, but I quite understood it by doing the following: $$\begin{split} y(x) &= w^Tx + w_0 \\&= w^T(x_{\perp} + r\frac{w}{\Vert w \Vert}) + w_0 \\&= w^T x_{\perp} + r\Vert w \Vert + w_0 \\&= w^T x_{\perp} + \frac{y(x)}{\Vert w \Vert}\Vert w \Vert + w_0 \\&= w^T x_{\perp} + y(x) + w_0 \end{split}$$ which means $w^T x_{\perp} + w_0 = 0$. Therefore, the point $x_{\perp}$ is such that $y(x_{\perp}) = 0$, therefore the red line. P.S: It got me confused because usually $x_{\perp}$ means a vector that nulls $x$, i.e. $x^T x_{\perp} = 0$, but this is not what you mean in the figure.
-First answer to: where did $\frac{y(x)}{\Vert w \Vert}$ come from ? Well, this is the distance between any point $x$ and the line $y(x_{\perp})$ (the red line). Using the following relation which gives the distance between point $x$ and line $y(x_{\perp})$, we have $$\text{dist}(y(x_{\perp}),x) = \frac{\vert y(x) \vert }{\Vert w \Vert}$$ where we can remove the $\vert \vert$ since $y(x)$ is positive in the figure above. [Reference, see https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line].
-Second answer to where did $-\frac{w_0}{\Vert w \Vert}$ come from: It is the distance between two parallel lines: The red line $y(x_{\perp})$ and the parallel black line say $z(x) = w^Tx$ since they have same directional vector $w$ and the black line passes thru the origin (so no intercept). Hence $$\text{dist}(y(x_{\perp}),z(x)) = \frac{\vert w_0 - 0 \vert }{\Vert w \Vert}$$ where the negative sign in the figure appears because it is towards the negative side. [Reference, see https://en.wikipedia.org/wiki/Distance_between_two_straight_lines].
• thx EL, I am trying to follow your answer. can you tell me how $$w^T(r * (w/||w||) )= r||w||$$ on the middle? – Mike Kim Oct 5 '16 at 11:35
• Yes so $w^T(r \frac{w}{\Vert w \Vert}) = r \frac{w^Tw}{\Vert w \Vert}$ since $r$ is a scalar. Also $w^Tw = \Vert w \Vert^2$. Therefore, $w^T(r \frac{w}{\Vert w \Vert}) = r \frac{w^Tw}{\Vert w \Vert} = r \frac{ \Vert w \Vert^2}{\Vert w \Vert} = r \Vert w \Vert$ – Ahmad Bazzi Oct 5 '16 at 14:45