According to Bishop's book, if your input is a multidimensional vector $x$, then non kernel based basis function models suffer from the curse of dimensionality:

y(x,w) be the linear regression function

But the above picture is based on polynomial basis functions in one dimension, $\phi_i(x)=x^i$.

If we had different basis functions, like sigmoidal $\phi(x)=\sigma((x-\mu_j)/s)$ where $\sigma(a)=(1/(1+exp(-a))$, would the sum still be equal to the above picture or would the combinations between the different dimensions differ?


You will always have as many terms in your sum, as you have basis-functions in your function space. Since you can combine univariate basis functions, you will not get out of it.

For example: You have monomials in $x$, $y$ and $z$ for degree in every direction up to 5. You will than have $3^6$ basis functions (1, $x$, $y$, $z$, $x^2$, ..., $x^5y^5z^4$, $x^5y^5z^5$), by using every chance of combining these to get new basis functions.

However you could use just a part of these. For example you decide, that the total degree matters. By that you will get less basis functions, less dimensionlity and less accuracy.

  • $\begingroup$ Thank you so much flr your answer! $\endgroup$ – Theodor Johnson Jan 19 '17 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.