Determining the line of best fit for a known function

Question

How can one determine the line of best fit (in the form $y=mx+b$) for a known function, such as $y=x^2$ over the domain $[1,2]$?

Normally I would use a least squares regression with a set of representative points from the function, but this application calls for a more rigorous analysis.

Edit: I'm having difficulties understanding the answers given, so I ended up digging in an old textbook to find the closed form solution of linear regression, then changing the summations to integrals.

Least squares regression guide: https://www.mathsisfun.com/data/least-squares-regression.html

Formulas with data points:
$m=(N*\sum(xy)-\sum(x)\sum(y))/(N*\sum(x^2)-(\sum(x))^2)$
$b=(\sum(y)-m*\sum(x))/N$

Answer 1

To approximate a function $f(x)$ from $a$ to $b$ using least squares, we want to minimize

$\begin{array}\\ D &=\int_a^b (f(x)-cx-d)^2dx\\ \text{so}\\ 0 &=\dfrac{\partial D}{\partial c}\\ &=\int_a^b (-2x)(f(x)-cx-d)dx\\ &=-2\int_a^b x(f(x)-cx-d)dx\\ &=-2\left(\int_a^b xf(x)dx-c\int_a^bx^2dx-\int_a^bdx dx\right)\\ &=-2\left(\int_a^b xf(x)dx-c\dfrac{b^3-a^3}{3}-d\dfrac{b^2-a^2}{2}\right)\\ \text{and}\\ 0 &=\dfrac{\partial D}{\partial d}\\ &=-\int_a^b (f(x)-cx-d)dx\\ &=-\int_a^b (f(x)-cx-d)dx\\ &=-\left(\int_a^b f(x)dx-c\int_a^bxdx-\int_a^bddx\right)\\ &=-\left(\int_a^b f(x)dx-c\dfrac{b^2-a^2}{2}-d(b-a)\right)\\ \end{array} $

This gives two equations in the two unknowns $c$ and $d$.

The determinant is

$\begin{array}\\ \dfrac{(b^2-a^2)^2}{4}-(b-a)\dfrac{b^3-a^3}{3} &=\dfrac{(b-a)^2}{12}(3(b+a)^2-4(b^2+ba+a^2))\\ &=\dfrac{(b-a)^2}{12}(3b^2+6ab+3a^2-4b^2-4ba-4a^2)\\ &=\dfrac{(b-a)^2}{12}(-b^2+2ba-a^2)\\ &=-\dfrac{(b-a)^4}{12}\\ \end{array} $

which, if $a \ne b$, is never zero, so the equations always have a unique solution.

Answer 2

What you want to do is find the projection of $y$ on to $\mathbb{P}^1[1,2]$. So you want to find a function $p(x)\in\mathbb{P}^1[1,2]$ such that $$ \int_{[1,2]}(y-p(x))q(x)\,dx=0\quad\forall\,q(x)\in\mathbb{P}^1[1,2].$$ This equates to finding a $p(x)=ax+b$ such that $$ \int_{[1,2]}y-p(x)\,dx=0\quad\textrm{and}\quad\int_{[1,2]}x(y-p(x))\,dx=0. $$ Note that this finds the closest linear function under the $L^2$-norm. Different inner-product spaces will find you different functions. However, the method above is the most intuitive and can be thought of as a continuous extension of a least squares fit (as the $L^2$-norm is the integral of the square).