# I need help understanding a few things regarding Least Squares Regression.

First of all, I imagine that I will get many downvotes on this since the questions are probably considered "stupid." However, please understand that I'm taking a class that is way above my head, and I'm not allowed to drop it. I'm asking these questions so that I know exactly what I need to learn without wasting time.

I'm taking a class in Big Data/Data Mining and I've been given an assignment to "do" least squares regression on a data set in Matlab. The problem is that I have never learned linear algebra before. I'm spending all my spare time catching up, but I feel like I will not make the deadline unless I focus only exactly on what the assignment requires.

I took notes during the classes, but I didn't know linear algebra back then so I couldn't understand what exactly I was writing. After spending weeks learning about vectors, matrices and subspaces, I understand some of it, but it's not enough.

Therefore, I ask you to help me understand the following:

1. If $x = [x(1), x(2), ... ,x(d)]$, then $\bar{x} = [1, x(1), x(2), ..., x(d)]$
(these are a column vectors, but I don't know the latex code).

However, I watched a Khan Academy video on linear regression where $\bar{x}$ was described as the mean value of $x$. From what I know, 'mean value' refers to a single (scalar) value and not a vector, so that has me confused. So, what does $\bar{x}$ actually mean?

2. The dependent value (I know the difference between independent and dependent values) is defined as $y(x) = w0 + w1x(1)+w2x(2)+...+ wdx(d)$

What on earth does $w$ mean? It's defined as a column vector ranging from $w0$ to $wd$.

3. What does $w*$ mean?

4. After doing the least squares regression, I'm asked to print the value of $w$ generated by the regression. Is this also a column vector or something completely different?

5. What am I actually trying to figure out with linear regression? A formula? A value? Nothing has been said about it in the class based on the notes I've taken, and I can't find any explanation in any book either. The closest I've gotten is that the objective is to "minimize the residuals." I understand what 'residual' means, but what is the output exactly?

I mean, I have training data. I use the regression formula on that data set. What am I left with that I can use on the test data?

6. Do I need to know anything about Eigen-values or anything involving "Eigen" in order to solve this problem?

Any help is greatly appreciated. Thank you.

• 3. (w*) is probably the transpose of the vector (w) but it depends on how it's defined in your class as (*) is used for many things. 5. The goal of linear regression to find the equation of a line which 'matches' your data the best. With this line you can predict values of (w) which you don't have data for. – TSF Oct 10 '17 at 9:22
• The transpose of the matrix is defined as $x^T$. Also, in the notes I've taken, there's the equation $x^Ty = x^Txw^*$. Edit: I'm not sure but I think it's actually $w^*$ and not $w*$ – Lobs001 Oct 10 '17 at 9:27
• @TonyS.F. I'm answering a second time on your comment. Isn't the goal to predict the $y$ value of $x$? I mean, if $x$ is the independent, and $y$ is the dependent. Where does $w$ come in exactly? edit: In my previous comment, I mean that the transpose of the vector is defined as $x^T$, not the matrix. – Lobs001 Oct 10 '17 at 9:31
• Yes you are correct, you want to predict the values of the dependent variable, my mistake. You are also correct that usually xbar is not a vector, but a scalar quantity representing the arithmetic average. It seems, though, that your text/class also uses xbar to denote an 'augemented vector' i.e. they put another entry with a '1' in it. – TSF Oct 10 '17 at 9:37
• Thanks! That's one mystery sort of solved. Although, I just googled 'augmented vector' and it seems like they put the '1' at the end/bottom and not at the top/beginning. Is that relevant? – Lobs001 Oct 10 '17 at 9:45

You are given a bunch of vectors $x_i \in \mathbb R^d$ and corresponding scalars $y_i \in \mathbb R$. Your goal is to find a vector $w \in \mathbb R^{d+1}$ such that $\bar x_i^T w \approx y_i$ for all $i$. We select $w$ to be the vector that minimizes the objective function $$E(w) = \sum_i (\bar x_i^T w - y_i)^2.$$ But minimizing $E(w)$ is something that we learned how to do in multivariable calculus. Just set the gradient of $E$ equal to $0$, and solve for $w$.
• @Lobs001 No, I don't mean that you should set $E(w)$ equal to $0$. In precalculus, you have learned to find the minimum point (vertex) on a parabola by the technique of completing the square. So, minimizing a quadratic function of one variable is something that you know how to do from precalculus. In calculus you will learn a different method for minimizing functions: you set the derivative equal to $0$. So, how do we minimize $E(w)$? Well, $E(w)$ is a quadratic function of several variables $w_0, w_1,\ldots, w_d$. You can either "complete the square" or "set the derivative to $0$". – littleO Oct 11 '17 at 3:35
• It is difficult to understand the details without knowing some multivariable calculus and linear algebra. Multivariable calculus and linear algebra are prerequisites for machine learning. Here is an overview: it is possible to rewrite $E(w)$ using matrix notation as $E(w) = \| X w - y \|^2$. Here $X$ is the matrix whose $i$th row is $\bar x_i^T$, and $y$ is the column vector whose $i$th component is $y_i$. From multivariable calculus, the gradient of $E$ is $\nabla E(w) = 2 X^T (Xw - y)$. Set the gradient of $E$ equal to $0$ and we get $X^T Xw = X^T y$. Now solve for $w$. – littleO Oct 11 '17 at 3:41
• The equations $X^T X w = X^T y$ are called the "normal equations". You can solve a least squares problem by writing down and then solving the normal equations. – littleO Oct 11 '17 at 3:42