I'm currently in a course learning about neural networks and machine learning, and I came across these two formulas in this textbook page on linear regression:

1) $y(x) = a + bx$


2) $y(x) = w^{T}\phi(x)$

What is the difference between these two formulas? How are they related? They seem to perform the same thing to me, so I can't figure out how to conceptually distinguish them, or when to use one over the other. I outline my understanding below.

My Understanding

Here's my current understanding (which likely is lacking) so that you can see how I'm viewing these equations.

$y(x) = w^{T}\phi(x)$ is representing the features being observed multiplied by their weights, this makes sense for learning because some features being observed could have a larger significance on what is being represented. My understanding is that $\phi(x^{i})$ is feature $i$, so the 10th feature would be represented as $\phi(x^{10})$.

$y(x) = a + bx$ however, seems to do the same thing to me (please correct me if I am wrong, I feel like this is incorrect). Both $a$ and $b$ are the trainable parameters, which get refined over several training sets in order to develop concrete weights so that $y(x)$ can perform accurate predictions about new values of $x$ that were not in the training set.

Main Question

What's further confusing about these two formulas is that they both use a least-difference of squares cost-function which needs to be minimized, making them seem even more similar. I would really like to explicitly separate these two formulas in my head, but I can't seem to determine how they differ.

  • $\begingroup$ I have not checked the textbook yet, but is it possible that $\phi(\mathbf{x})$ describes a function of the observed features, in order for example to capture nonlinear relationships? $\endgroup$ – megas Jan 15 '15 at 4:31
  • $\begingroup$ I checked the textbook (it finally loaded). (1) is a special case of (2). The second case captures a more general dependence between $y(x)$ and $x$: $y(x)$ can be linear function (not of $x$ directly, but) of some transformation (function $\phi$) of $x$. For example, you observe $x$ and $y(x)$ may not depend linearly on $x$. But you might be able to create some new features $\phi(x)$ (based on $x$) and then $y(x)$ appears to be a linear function of the new (artificially created) features. The book has an example with fitting a cubic polynomial. $\endgroup$ – megas Jan 15 '15 at 4:40
  • $\begingroup$ My apologies about the textbook, my professor sent me the link as well and it loaded quite slowly. Ahh it's a special case, I'll read your answer in more depth, thanks so much for the insight! $\endgroup$ – CisBae Jan 15 '15 at 5:32

(1) is a special case of (2). Relation (2) captures a more general dependence between $y(\mathbf{x})$ and $\mathbf{x}$: it allows $y(\mathbf{x})$ to be a linear function (not of $\mathbf{x}$ directly, but) of some transformation (function $\phi(\cdot)$) of $\mathbf{x}$.

For example, you may have observations $\left( \mathbf{x}, y(\mathbf{x})\right)$, where $y(\mathbf{x})$ does not depend linearly on $\mathbf{x}$. But it might depend linearly on some transformation of $\mathbf{x}$, i.e, on some new feature vector $\phi(\mathbf{x})$ that is created by some (possibly non-linear) transformation of $\mathbf{x}$.

The book has an example with fitting a cubic polynomial. You observe $\left(x, y(x)\right)$ (note that $x$ here happens to be a scalar) and $y(x)$ is a cubic polynomial of $x$: $y(x) = a+bx+cx^2+dx^3$. You know this relationship because either an oracle revealed it to you, or because you saw some plot of your data and you could figure it out based on your experience. Then you still need to find the coefficients for this polynomial. But $y(x)$ which is a cubic polynomial function of $x$, is just a linear function of the vector $\phi(x) = (1, x, x^2, x^3)$. In this example, $\phi(x)$ is a vector of dimension $4$, produced by the single observed feature $x$. You can think of $\phi(x)$ as a new type of feature. Now you can use linear regression to find the desired coefficients of the cubic polynomial.

Finally, note that the book also explains why (1) is a special case of (2): you can just assume that $\phi( \mathbf{x}) = (1, \mathbf{x})$, i.e., a vector obtained by concatenating $ \mathbf{x}$ with $1$.

| cite | improve this answer | |
  • $\begingroup$ That clears up a lot, I think I understand it now. I'll read more into it as well and try finding the coefficients of the cubic polynomial example in the textbook, thanks again! Very good answer. $\endgroup$ – CisBae Jan 15 '15 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.