# Formulating the polynomial regression

I'm trying to formulate a regression problem such that $$y=ax^{b}$$. Previously, I formulated the $$y=ax+b$$ like $$y=Ac+e$$ where $$c= \begin{bmatrix} a\\ b \end{bmatrix}$$ and $$A= \begin{bmatrix} x_1 & 1\\ x_2 & 1\\ . & \\ . & \\ x_n & 1 \end{bmatrix}$$

Can I only change the A matrix such that $$A= \begin{bmatrix} x_1^{b} & 1\\ x_2^{b} & 1\\ . & \\ . & \\ x_n^{b} & 1 \end{bmatrix}$$ and change c as $$c= \begin{bmatrix} a\\ 0 \end{bmatrix}$$ and write the same formula $$y=Ac+e$$

Am I correct? Any help would be appreciated. Many thanks

• To use linear regression, your model has to be linear in the parameters. Your model is not linear in $a$ and $b$. – John Douma Mar 14 at 15:59
• If $a$ and $b$ are to be determined you probably might want to consider the model $\ln y= b \ln x+\ln a$, a linear model quite frequently used for powerfunctions. – Jens Schwaiger Mar 14 at 16:57

No. You need all the powers in the columns of A.

Write the problem as minimizing $$D=\sum_{k=1}^m (y_k-\sum_{j=0}^n a_jx_k^j)^2$$.

Set $$\partial D/\partial a_j =0$$ for each $$j$$ and see what you get.

• Sir, when I take the derivative, I get the following equation: $$y_k=\sum_{j=0}^{n} a_j x_k^{j}$$ How can I proceed to express them in matrix form? Many thanks – Jason Mar 14 at 16:11
• \begin{bmatrix} y1 \\ y2 \\ . \\ . \\ y_n \end{bmatrix} = \begin{bmatrix} a & 0 & 0 & ... & 0 \\ a & a & 0 & ... & 0 \\ . \\ . \\ a & a & a & ... & a \end{bmatrix}\begin{bmatrix} x_1^{b} \\ x_2^{b} \\ . \\ . \\ x_n^{b} \end{bmatrix} + \begin{bmatrix} e_1 \\ e_2 \\ . \\ . \\ e_n \end{bmatrix} Am I correct now, sir? – Jason Mar 14 at 16:37

You have three alternatives to solve this problem.

1. Alternative: As Jens Schwaiger proposed you can rewrite the equation as $$\ln y_i = \ln a + b \ln x_i+ \tilde{\varepsilon}_i$$. If we introduce the coefficient $$\tilde{a}=\ln a$$ and the transformed outputs $$\tilde{y}_i=\ln y_i$$ then it is possible to perform a standard linear regression. the coefficients $$\boldsymbol{w}=[\tilde{a},b]^T=[\ln a, b]^T$$ can be estimated by the least squares estimate

$$\hat{\boldsymbol{w}}=\left[\boldsymbol{\Phi}^T\boldsymbol{\Phi} \right]^{-1}\boldsymbol{\Phi}^T\tilde{\boldsymbol{y}},\qquad (*)$$

in which $$\tilde{\boldsymbol{y}}=\left[\ln y_1, \ln y_2, \ldots, \ln y_n\right]^T$$ is the transformed output vector and

$$\boldsymbol{\Phi} = \begin{bmatrix}1 & \ln x_1\\1 & \ln x_2 \\ \vdots & \vdots \\1 & \ln x_n \end{bmatrix}.$$

2. Alternative: We have $$y_i = ax_i^{b}+\varepsilon_i=a\exp(b\ln x_i)+\varepsilon_i$$. We can now expand the exponential function by a taylor series.

$$y_i=a\left[1+b\ln x_i + b^2/2!\left[\ln x_i\right]^2+b^3/3![\ln x_i]^3+\cdots \right]+\varepsilon_i$$

If we truncate the series to the $$m^{\text{th}}$$ power then we can approximate $$y_i$$ as

$$y_i \approx \left[a+ab\ln x_i + ab^2/2!\left[\ln x_i\right]^2+ab^3/3![\ln x_i]^3+\cdots +ab^m/m!\left[\ln x_i \right]^m\right]+\varepsilon_i.$$

By introducing the coefficients $$w_l = ab^l/l!$$ we can rewrite the previous equation as

$$y_i \approx \left[w_0+w_1\ln x_i + w_2\left[\ln x_i\right]^2+w_3[\ln x_i]^3+\cdots +w_m\left[\ln x_i \right]^m\right]+\varepsilon_i.$$

The coefficients are given by equation $$(*)$$ but now $$\tilde{\boldsymbol{y}}=\boldsymbol{y}=\left[y_1, y_2, \ldots,y_n \right]^T$$ and

$$\boldsymbol{\Phi} = \begin{bmatrix}1 & \ln x_1 & [\ln x_1]^2 & \cdots & [\ln x_1]^m\\1 & \ln x_2 & [\ln x_2]^2 & \cdots & [\ln x_2]^m\\ \vdots & \vdots & \vdots & \ddots & \vdots \\1 & \ln x_n & [\ln x_n]^2 & \cdots & [\ln x_n]^m\\\end{bmatrix}.$$

After having obtained the coefficients $$\boldsymbol{w}=[a, ab, ab^2, ..., ab^m]$$ you can determine $$a=w_0$$ and $$b=w_1/w_0$$ and so forth.

3. Alternative: Full nonlinear least squares (proposed by Marty Cohen), which does not have a closed form solution. Here we use the objective function $$E(\boldsymbol{w}=[a,b]^T)$$ which is

$$E(\boldsymbol{w})=\sum_{i=1}^{n}[y_i-ax_i^b]^2.$$

The partial derivatives are given by

$$\dfrac{\partial E}{\partial a} = \sum_{i=1}^n2[y_i-ax_i^b](-x_i^b)$$ $$\dfrac{\partial E}{\partial b} = \sum_{i=1}^n2[y_i-ax_i^b](-ax_i^b\ln x_i).$$

After setting these partial derivatives equal to zero you will have to solve a nonlinear equation in the coefficients $$a$$ and $$b$$. You can try to numerically solve this equation by using Newton-Raphson.