So, I'm making a simple program for drawing graphs, and I'm looking at making some simple best-fit curves using some basic regression analysis. I've happily got linear and quadratic regression working (thanks to this post), but it's not quite detailed enough. I'm aware that cubic curves can be extremely good at this, within reason (and hence why certain spline methods are constructed with them), so I've attempted to expand this into a cubic form, but it doesn't seem to work at all. I've looked around the internet for several hours, and found pretty much exclusively posts about how to do this with a graphic calculator, or a program such as MATLAB. Obviously, this isn't much help, since I'm trying to implement it from scratch. I tried using some of the general forms available on the Wikipedia page for regression analysis, but they gave me an incredibly steep curve that was wildly off from the data presented. I'm sure I'm doing something wrong, but could someone help me to discover what exactly that is?
1 Answer
I'm not a statistician, but here is a simple least squares implementation that works for me:
Suppose you have data: $(t_1,y_1), \ldots (t_n,y_n)$. So $t_i$ is in each case the independent parameter and $y_i$ the (measured) value at such a data point. Construct a matrix: \begin{equation}A=\begin{bmatrix} t_1^3 & t_1^2 & t_1 & 1 \\ \vdots & \vdots & \vdots & \vdots \\ t_n^3 & t_n^2 & t_n & 1 \end{bmatrix}\end{equation} and let \begin{equation}y=\begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix}. \end{equation} Then \begin{equation} x=(A^*A)^{-1}A^*y=\begin{bmatrix}a\\b\\c\\d \end{bmatrix} \end{equation} is the least squares solution: $y_0=at_0^3+bt_0^2+ct_0+d$. The error $E$ may be computed as $\|Ax-y\|^2$.
The source for my method is a textbook on linear algebra: Linear Algebra, 4th edition by Friedberg et al. p.360-364. It is really used in the textbook just as a demonstration of some practical applications in terms of the theory of inner product spaces, but ever since I came across this, and when I needed to do basic regression, this has worked well for me in the case of polynomial fits.
-
$\begingroup$ so btw, as you can see, it is easy to modify this method to do a polynomial fit of any degree you desire... $\endgroup$ Apr 5, 2014 at 20:40
-
$\begingroup$ the only condition is that the matrix $A$ must be rank $d+1$, where $d$ is the polynomial degree, else $A^*A$ is not invertible. $\endgroup$ Apr 5, 2014 at 20:51
-
$\begingroup$ I'm assuming A* here is the transpose? Because I tried this, and for some reason it diverges ridiculously quickly (to the point of crashing due to floating point overflow). Not sure what I'm doing wrong... $\endgroup$– HoeloeApr 5, 2014 at 20:52
-
$\begingroup$ Yes $A^*$ is the transpose...if you are just using real data. I'd suggest using a small data set to start with. How are you inverting the matrix $A^*A$? $\endgroup$ Apr 5, 2014 at 20:56
-
$\begingroup$ I tried running it through WolframAlpha and copying the result (from a slightly different form of the equation), and also using an open source Matrix library, and using the Invert function that was built in. This image is from the matrix library attempt, with the quadratic fit plotted for comparison: i.imgur.com/2ijDUim.png $\endgroup$– HoeloeApr 5, 2014 at 21:25
