0
$\begingroup$

First of all, I imagine that I will get many downvotes on this since the questions are probably considered "stupid." However, please understand that I'm taking a class that is way above my head, and I'm not allowed to drop it. I'm asking these questions so that I know exactly what I need to learn without wasting time.

I'm taking a class in Big Data/Data Mining and I've been given an assignment to "do" least squares regression on a data set in Matlab. The problem is that I have never learned linear algebra before. I'm spending all my spare time catching up, but I feel like I will not make the deadline unless I focus only exactly on what the assignment requires.

I took notes during the classes, but I didn't know linear algebra back then so I couldn't understand what exactly I was writing. After spending weeks learning about vectors, matrices and subspaces, I understand some of it, but it's not enough.

Therefore, I ask you to help me understand the following:

1. If $x = [x(1), x(2), ... ,x(d)]$, then $\bar{x} = [1, x(1), x(2), ..., x(d)] $
(these are a column vectors, but I don't know the latex code).

However, I watched a Khan Academy video on linear regression where $\bar{x}$ was described as the mean value of $x$. From what I know, 'mean value' refers to a single (scalar) value and not a vector, so that has me confused. So, what does $\bar{x}$ actually mean?

2. The dependent value (I know the difference between independent and dependent values) is defined as $y(x) = w0 + w1x(1)+w2x(2)+...+ wdx(d)$

What on earth does $w$ mean? It's defined as a column vector ranging from $w0$ to $wd$.

3. What does $w*$ mean?

4. After doing the least squares regression, I'm asked to print the value of $w$ generated by the regression. Is this also a column vector or something completely different?

5. What am I actually trying to figure out with linear regression? A formula? A value? Nothing has been said about it in the class based on the notes I've taken, and I can't find any explanation in any book either. The closest I've gotten is that the objective is to "minimize the residuals." I understand what 'residual' means, but what is the output exactly?

I mean, I have training data. I use the regression formula on that data set. What am I left with that I can use on the test data?

6. Do I need to know anything about Eigen-values or anything involving "Eigen" in order to solve this problem?

Any help is greatly appreciated. Thank you.

$\endgroup$
  • $\begingroup$ 3. (w*) is probably the transpose of the vector (w) but it depends on how it's defined in your class as (*) is used for many things. 5. The goal of linear regression to find the equation of a line which 'matches' your data the best. With this line you can predict values of (w) which you don't have data for. $\endgroup$ – TSF Oct 10 '17 at 9:22
  • $\begingroup$ The transpose of the matrix is defined as $x^T$. Also, in the notes I've taken, there's the equation $x^Ty = x^Txw^*$. Edit: I'm not sure but I think it's actually $w^*$ and not $w*$ $\endgroup$ – Lobs001 Oct 10 '17 at 9:27
  • $\begingroup$ @TonyS.F. I'm answering a second time on your comment. Isn't the goal to predict the $y$ value of $x$? I mean, if $x$ is the independent, and $y$ is the dependent. Where does $w$ come in exactly? edit: In my previous comment, I mean that the transpose of the vector is defined as $x^T$, not the matrix. $\endgroup$ – Lobs001 Oct 10 '17 at 9:31
  • $\begingroup$ Yes you are correct, you want to predict the values of the dependent variable, my mistake. You are also correct that usually xbar is not a vector, but a scalar quantity representing the arithmetic average. It seems, though, that your text/class also uses xbar to denote an 'augemented vector' i.e. they put another entry with a '1' in it. $\endgroup$ – TSF Oct 10 '17 at 9:37
  • $\begingroup$ Thanks! That's one mystery sort of solved. Although, I just googled 'augmented vector' and it seems like they put the '1' at the end/bottom and not at the top/beginning. Is that relevant? $\endgroup$ – Lobs001 Oct 10 '17 at 9:45
0
$\begingroup$

You are given a bunch of vectors $x_i \in \mathbb R^d$ and corresponding scalars $y_i \in \mathbb R$. Your goal is to find a vector $w \in \mathbb R^{d+1}$ such that $\bar x_i^T w \approx y_i$ for all $i$. We select $w$ to be the vector that minimizes the objective function $$ E(w) = \sum_i (\bar x_i^T w - y_i)^2. $$ But minimizing $E(w)$ is something that we learned how to do in multivariable calculus. Just set the gradient of $E$ equal to $0$, and solve for $w$.

$\endgroup$
  • $\begingroup$ Thanks! I have basically the same formula in my notes, but your explanation made it make sense. Oh, one more thing. I've only taken pre-calculus. When you say "set the gradient of E equal to 0," do you mean that I should just set E(w) equal to 0? If not, what is the gradient of E exactly? $\endgroup$ – Lobs001 Oct 11 '17 at 2:43
  • $\begingroup$ @Lobs001 No, I don't mean that you should set $E(w)$ equal to $0$. In precalculus, you have learned to find the minimum point (vertex) on a parabola by the technique of completing the square. So, minimizing a quadratic function of one variable is something that you know how to do from precalculus. In calculus you will learn a different method for minimizing functions: you set the derivative equal to $0$. So, how do we minimize $E(w)$? Well, $E(w)$ is a quadratic function of several variables $w_0, w_1,\ldots, w_d$. You can either "complete the square" or "set the derivative to $0$". $\endgroup$ – littleO Oct 11 '17 at 3:35
  • $\begingroup$ Ah, that makes sense. Thank you! I should be able to figure out the rest by myself now. $\endgroup$ – Lobs001 Oct 11 '17 at 3:40
  • $\begingroup$ It is difficult to understand the details without knowing some multivariable calculus and linear algebra. Multivariable calculus and linear algebra are prerequisites for machine learning. Here is an overview: it is possible to rewrite $E(w)$ using matrix notation as $E(w) = \| X w - y \|^2$. Here $X$ is the matrix whose $i$th row is $\bar x_i^T$, and $y$ is the column vector whose $i$th component is $y_i$. From multivariable calculus, the gradient of $E$ is $\nabla E(w) = 2 X^T (Xw - y)$. Set the gradient of $E$ equal to $0$ and we get $X^T Xw = X^T y$. Now solve for $w$. $\endgroup$ – littleO Oct 11 '17 at 3:41
  • $\begingroup$ The equations $X^T X w = X^T y$ are called the "normal equations". You can solve a least squares problem by writing down and then solving the normal equations. $\endgroup$ – littleO Oct 11 '17 at 3:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.