# Linear regression of basis functions for multivariate inputs

Background - My current understanding of linear regression of basis functions:

Given an input domain $$\mathcal{X}$$, target domain $$\mathcal{Y}$$, and a data set $$S=\left\{ \left(x_{i},y_{i}\right)\right\} _{i=1}^{m}$$. Assuming that for each $$i\ \$$, $$y_{i}=g\left(x_{i}\right)+\xi$$ where $$\xi\sim\mathcal{N}\left(0,\sigma\right)$$ is a noise term, we would like to estimate $$g$$ from $$S$$. From here forth we identify $$g$$ with some parameter vector $$w$$.

It turns out that in this scenario the best $$w$$ w.r.t. squared loss $$\hat{w}_{\ell_{2}}=\underset{w}{\arg\min}\sum_{i=1}^{m}\left\Vert g_{w}\left(x_{i}\right)-y_{i}\right\Vert ^{2}$$

and the maximum likelihood estimator

$$\hat{w}_{ML}=\underset{w}{\arg\max}\left[\ell\left(g_{w};S\right)\right]=\underset{w}{\arg\max}\left[\log p\left(y^{\left(n\right)}\mid x^{\left(n\right)},w\right)\right]$$

are in fact the same.

Finding $$\hat{w}_{ML}$$ can be intractable if $$g$$ is non linear. However, the problem becomes easy if $$g$$ is linear in some set of basis functions. $$g_{w}\left(x\right)=\left\langle w\mid\phi\left(x\right)\right\rangle =\sum_{j=1}^{d}w_{j}\phi_{j}\left(x\right)$$

(assuming $$x_{i}\in\mathbb{R}$$) since in this case, if our original data patterns were $$x_{1}\ldots x_{m}$$, we can map them to the following matrix $$M$$ $$\begin{array}{c} \left(\begin{array}{c} x_{1}\\ \vdots\\ x_{m} \end{array}\right)\mapsto\overbrace{\left(\begin{array}{ccc} \phi_{1}\left(x_{1}\right) & \cdots & \phi_{d}\left(x_{1}\right)\\ \vdots & & \vdots\\ \phi_{1}\left(x_{m}\right) & \cdots & \phi_{d}\left(x_{m}\right) \end{array}\right)}^{M}\end{array}$$

and then this becomes a linear regression problem (minimize $$\left\Vert y-Mw\right\Vert ^{2}$$), Since, $$Mw=\begin{array}{c} \left(\begin{array}{ccc} \phi_{1}\left(x_{1}\right) & \cdots & \phi_{d}\left(x_{1}\right)\\ \vdots & & \vdots\\ \phi_{1}\left(x_{m}\right) & \cdots & \phi_{d}\left(x_{m}\right) \end{array}\right)\left(\begin{array}{c} w_{1}\\ \vdots\\ w_{d} \end{array}\right)=\end{array}\left(\begin{array}{c} g_{w}\left(x_{1}\right)\\ \vdots\\ g_{w}\left(x_{m}\right) \end{array}\right)$$

Question:

Is this scheme, of mapping the inputs via some set of basis functions, generalizable to the case where the inputs $$x_{i}$$ are multidimensional? This seems similar to the idea of kernel functions, only that we are dealing with regression here rather than binary classification.

• Do you mean $x_i \in \mathbb{R}^N$? If yes, then it's possible and check, e.g., Bishop's book. – user550103 Jan 11 at 5:27