Background - My current understanding of linear regression of basis functions:

Given an input domain $\mathcal{X}$, target domain $\mathcal{Y}$, and a data set $S=\left\{ \left(x_{i},y_{i}\right)\right\} _{i=1}^{m}$. Assuming that for each $i\ \ $, $y_{i}=g\left(x_{i}\right)+\xi$ where $\xi\sim\mathcal{N}\left(0,\sigma\right)$ is a noise term, we would like to estimate $g$ from $S$. From here forth we identify $g$ with some parameter vector $w$.

It turns out that in this scenario the best $w$ w.r.t. squared loss $$\hat{w}_{\ell_{2}}=\underset{w}{\arg\min}\sum_{i=1}^{m}\left\Vert g_{w}\left(x_{i}\right)-y_{i}\right\Vert ^{2}$$

and the maximum likelihood estimator

$$\hat{w}_{ML}=\underset{w}{\arg\max}\left[\ell\left(g_{w};S\right)\right]=\underset{w}{\arg\max}\left[\log p\left(y^{\left(n\right)}\mid x^{\left(n\right)},w\right)\right]$$

are in fact the same.

Finding $\hat{w}_{ML}$ can be intractable if $g$ is non linear. However, the problem becomes easy if $g$ is linear in some set of basis functions. $$g_{w}\left(x\right)=\left\langle w\mid\phi\left(x\right)\right\rangle =\sum_{j=1}^{d}w_{j}\phi_{j}\left(x\right)$$

(assuming $x_{i}\in\mathbb{R}$) since in this case, if our original data patterns were $x_{1}\ldots x_{m}$, we can map them to the following matrix $M$ $$\begin{array}{c} \left(\begin{array}{c} x_{1}\\ \vdots\\ x_{m} \end{array}\right)\mapsto\overbrace{\left(\begin{array}{ccc} \phi_{1}\left(x_{1}\right) & \cdots & \phi_{d}\left(x_{1}\right)\\ \vdots & & \vdots\\ \phi_{1}\left(x_{m}\right) & \cdots & \phi_{d}\left(x_{m}\right) \end{array}\right)}^{M}\end{array}$$

and then this becomes a linear regression problem (minimize $\left\Vert y-Mw\right\Vert ^{2}$), Since, $$Mw=\begin{array}{c} \left(\begin{array}{ccc} \phi_{1}\left(x_{1}\right) & \cdots & \phi_{d}\left(x_{1}\right)\\ \vdots & & \vdots\\ \phi_{1}\left(x_{m}\right) & \cdots & \phi_{d}\left(x_{m}\right) \end{array}\right)\left(\begin{array}{c} w_{1}\\ \vdots\\ w_{d} \end{array}\right)=\end{array}\left(\begin{array}{c} g_{w}\left(x_{1}\right)\\ \vdots\\ g_{w}\left(x_{m}\right) \end{array}\right)$$


Is this scheme, of mapping the inputs via some set of basis functions, generalizable to the case where the inputs $x_{i}$ are multidimensional? This seems similar to the idea of kernel functions, only that we are dealing with regression here rather than binary classification.

  • $\begingroup$ Do you mean $x_i \in \mathbb{R}^N$? If yes, then it's possible and check, e.g., Bishop's book. $\endgroup$ – user550103 Jan 11 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.