I'm trying to formulate a regression problem such that $y=ax^{b}$. Previously, I formulated the $y=ax+b$ like $y=Ac+e$ where $c= \begin{bmatrix} a\\ b \end{bmatrix}$ and $A= \begin{bmatrix} x_1 & 1\\ x_2 & 1\\ . & \\ . & \\ x_n & 1 \end{bmatrix}$

Can I only change the A matrix such that $A= \begin{bmatrix} x_1^{b} & 1\\ x_2^{b} & 1\\ . & \\ . & \\ x_n^{b} & 1 \end{bmatrix}$ and change c as $c= \begin{bmatrix} a\\ 0 \end{bmatrix}$ and write the same formula $y=Ac+e$

Am I correct? Any help would be appreciated. Many thanks

  • $\begingroup$ To use linear regression, your model has to be linear in the parameters. Your model is not linear in $a$ and $b$. $\endgroup$ – John Douma Mar 14 at 15:59
  • 1
    $\begingroup$ If $a$ and $b$ are to be determined you probably might want to consider the model $\ln y= b \ln x+\ln a$, a linear model quite frequently used for powerfunctions. $\endgroup$ – Jens Schwaiger Mar 14 at 16:57

No. You need all the powers in the columns of A.

Write the problem as minimizing $D=\sum_{k=1}^m (y_k-\sum_{j=0}^n a_jx_k^j)^2$.

Set $\partial D/\partial a_j =0$ for each $j$ and see what you get.

  • $\begingroup$ Sir, when I take the derivative, I get the following equation: $$y_k=\sum_{j=0}^{n} a_j x_k^{j} $$ How can I proceed to express them in matrix form? Many thanks $\endgroup$ – Jason Mar 14 at 16:11
  • $\begingroup$ \begin{bmatrix} y1 \\ y2 \\ . \\ . \\ y_n \end{bmatrix} = \begin{bmatrix} a & 0 & 0 & ... & 0 \\ a & a & 0 & ... & 0 \\ . \\ . \\ a & a & a & ... & a \end{bmatrix}\begin{bmatrix} x_1^{b} \\ x_2^{b} \\ . \\ . \\ x_n^{b} \end{bmatrix} + \begin{bmatrix} e_1 \\ e_2 \\ . \\ . \\ e_n \end{bmatrix} Am I correct now, sir? $\endgroup$ – Jason Mar 14 at 16:37

You have three alternatives to solve this problem.

1. Alternative: As Jens Schwaiger proposed you can rewrite the equation as $\ln y_i = \ln a + b \ln x_i+ \tilde{\varepsilon}_i$. If we introduce the coefficient $\tilde{a}=\ln a$ and the transformed outputs $\tilde{y}_i=\ln y_i$ then it is possible to perform a standard linear regression. the coefficients $\boldsymbol{w}=[\tilde{a},b]^T=[\ln a, b]^T$ can be estimated by the least squares estimate

$$\hat{\boldsymbol{w}}=\left[\boldsymbol{\Phi}^T\boldsymbol{\Phi} \right]^{-1}\boldsymbol{\Phi}^T\tilde{\boldsymbol{y}},\qquad (*)$$

in which $\tilde{\boldsymbol{y}}=\left[\ln y_1, \ln y_2, \ldots, \ln y_n\right]^T$ is the transformed output vector and

$$\boldsymbol{\Phi} = \begin{bmatrix}1 & \ln x_1\\1 & \ln x_2 \\ \vdots & \vdots \\1 & \ln x_n \end{bmatrix}.$$

2. Alternative: We have $y_i = ax_i^{b}+\varepsilon_i=a\exp(b\ln x_i)+\varepsilon_i$. We can now expand the exponential function by a taylor series.

$$y_i=a\left[1+b\ln x_i + b^2/2!\left[\ln x_i\right]^2+b^3/3![\ln x_i]^3+\cdots \right]+\varepsilon_i $$

If we truncate the series to the $m^{\text{th}}$ power then we can approximate $y_i$ as

$$y_i \approx \left[a+ab\ln x_i + ab^2/2!\left[\ln x_i\right]^2+ab^3/3![\ln x_i]^3+\cdots +ab^m/m!\left[\ln x_i \right]^m\right]+\varepsilon_i.$$

By introducing the coefficients $w_l = ab^l/l!$ we can rewrite the previous equation as

$$y_i \approx \left[w_0+w_1\ln x_i + w_2\left[\ln x_i\right]^2+w_3[\ln x_i]^3+\cdots +w_m\left[\ln x_i \right]^m\right]+\varepsilon_i.$$

The coefficients are given by equation $(*)$ but now $\tilde{\boldsymbol{y}}=\boldsymbol{y}=\left[y_1, y_2, \ldots,y_n \right]^T$ and

$$\boldsymbol{\Phi} = \begin{bmatrix}1 & \ln x_1 & [\ln x_1]^2 & \cdots & [\ln x_1]^m\\1 & \ln x_2 & [\ln x_2]^2 & \cdots & [\ln x_2]^m\\ \vdots & \vdots & \vdots & \ddots & \vdots \\1 & \ln x_n & [\ln x_n]^2 & \cdots & [\ln x_n]^m\\\end{bmatrix}.$$

After having obtained the coefficients $\boldsymbol{w}=[a, ab, ab^2, ..., ab^m]$ you can determine $a=w_0$ and $b=w_1/w_0$ and so forth.

3. Alternative: Full nonlinear least squares (proposed by Marty Cohen), which does not have a closed form solution. Here we use the objective function $E(\boldsymbol{w}=[a,b]^T)$ which is


The partial derivatives are given by

$$\dfrac{\partial E}{\partial a} = \sum_{i=1}^n2[y_i-ax_i^b](-x_i^b)$$ $$\dfrac{\partial E}{\partial b} = \sum_{i=1}^n2[y_i-ax_i^b](-ax_i^b\ln x_i).$$

After setting these partial derivatives equal to zero you will have to solve a nonlinear equation in the coefficients $a$ and $b$. You can try to numerically solve this equation by using Newton-Raphson.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.