# Extremely poor polyfit, what am I doing wrong?

I have a dataset with me: http://pastebin.com/YZArky1j, which I am trying to polyfit. This is what I used to perform the polyfit:

x = [i[0] for i in data] # Get all x co-ordinates
y = [i[1] for i in data] # Get all y co-ordinates

coeff = polyfit(x, y, 5)


I am using the polyfit function from numpy: http://docs.scipy.org/doc/numpy/reference/generated/numpy.polyfit.html

The coefficients I get are: $$-2.84904034523 \times 10^{15} \\ 2.46136213423 \times 10^{11} \\ 8.66716249115 \times 10^{6} \\ -5.3534845897 \times 10^{2} \\ -0.577304577155 \times 10^{-1} \\ 2.54464720615 \times 10^{-6} \\$$

The plot of the polynomial with the plot of data looks like:

Here, red is the polynomial function and the blue is a plot of the data. So far, it is looking really good. This data is actually the prize pool of a gaming tournament v/s the day from when it started. So, when I try to project as to how it would look when the tournament actually starts (and there will be no more increase to the prize pool), I get a ridiculous number: $498,527,857.5$. This absolutely cannot be the case since the pool right now is around 6 million, there is no way that in two months it will reach 400 million+ in the remaining 2 months, especially as a look at the graph tells us that the rate of rate of growth is decreasing.

So, maybe I should try to fit it in the form $y = B + log(x)*A$. So, I tried to find the values of A and B:

log_x = [math.log(i) for i in x]
coeff_log = polyfit(log_x, y, 1)


And the coefficients I get are:

$$5.28101367 \times 10^{9} \\ -5.11847659 \times 10^{10}$$

And now, the projected prize pool looks something like: $24,987,469$. This also is highly suspect and I don't think is the correct projection.

What am I missing or what is it that I am doing wrong?

• Have you tried a logistic fit?en.wikipedia.org/wiki/Logistic_regression May 22, 2014 at 6:48
• 1) Once one move away from the range of data, the quality of a polyfit based extrapolation can deteriorate pretty quickly. Don't use it. 2) If you make a plot of $\log x$ for $x$ over $[16200,1621x]$, you will notice the graph is pretty straight there. Your result is no different from doing a linear fit. Try fitting your data with something like $A\log(x - \alpha) + B$ for an $\alpha \sim 16200$ instead. May 22, 2014 at 7:06
• Using a polynomial fit is OK as long as you just plan to interpolate. Never use it if you need extrapolation. Did you try what Achille Hui proposes ? May 22, 2014 at 7:27
• You can estimate $\alpha$ by minimizing the mean square error of $y - (A\log(x-\alpha) + B)$. If I didn't make any mistake, $\alpha \sim 16198.19255$. For this $\alpha$, the corresponding $A = 2.0671256 \times 10^6$, $B = 0.9049153 \times 10^6$. At $x = 16260$, I get something like $y \sim 9.4163695 \times 10^6$. May 22, 2014 at 7:58
• Thank you so much for that! I did try with what you mentioned: pastebin.com/EVf5efPd . And this is the plot I got: imgur.com/0xCOVRF . And the value for just pre-day prize pool (July 17th) is around 7.62 million. Which is something that doesn't sound very wrong (because the growth rate has been falling so rapidly, first 5 million was raised in around a week, but then it took another week to reach till 6 million). Right now I am trying to arrive at the value of optional $\alpha$ of 16260 as you mentioned. Thanks again for the help! May 23, 2014 at 3:57

Since I need to post a picture, I will expand some of my comments and turn them into an answer.

• Once one move away from the range of data, the quality of a polyfit based extrapolation can deteriorate pretty quickly. Don't use it.

• If you make a plot of $\log x$ vs $x$ over $[16200,16212]$, you will notice the graph is pretty straight there. Your result is no different from doing a linear fit.

It is understandable why you want to use $\log x$ for fitting the data. In fact, it is a good choice to capture the concavity of the provided data. However, to use a logarithm fit, one need to offset $x$ to the point where the data supposed to kick start. i.e one should try a fit of the form:

$$y = y(x) = A \log(x - \alpha) + B\quad\text{ where }\quad \alpha \sim 16200.$$

The next issues is how to fix/determine $\alpha$, $A$ and $B$. Instead of a full scale non-linear least square fit, one can

1. treat $\alpha$ as a parameter, perform a linear least square fit to obtain $A$ and $B$.
2. compute the mean standard error for this $\alpha$ once $A$ and $B$ are known.
3. determine $\alpha$ by minimizing the mean squared error.

I'm not going to derive anything. I'll just write down some formula so that you can try the way I perform the fit.

• Let $N$ be the number of samples
• Let $x_i$ be the date number for the $i^{th}$ sample.
(in unit date/time format, date 0 = 1970/01/01 mid-night)
• Let $y_i$ be the prize pool in unit of millions.

For any two vectors $X_i$, $Y_i$ of size $N$, define $$\begin{cases} \langle X \rangle &= \frac{1}{N}\sum_{i=1}^{N} X_i\\ \text{Corr}( X, Y ) &= \langle XY \rangle - \langle X\rangle\langle Y\rangle\\ \text{Var}(X) &= \langle X^2 \rangle - \langle X \rangle^2 \end{cases}$$ The $A$ and $B$ which minimize the MSE (mean square error):

$$\text{MSE}(X,Y) = \langle ( Y - A X - B )^2 \rangle = \frac{1}{N}\sum_{i=1}^N (Y_i - A X_i - B)^2$$

is given by $\displaystyle\;\begin{cases} A_{opt} &= \frac{\text{Corr}(X,Y)}{\text{Var}(X)}\\ B_{opt} &= \langle Y \rangle - A \langle X \rangle \end{cases}$. The corresponding optimal MSE also has a simple formula

$$\text{MSE}_{opt}(X,Y) = \text{Var}(Y) - \frac{\text{Corr}(X,Y)^2}{\text{Var}(X)}$$

To fix $\alpha$ for this problem, one just need to find the $\alpha$ which minimize $\;\text{MSE}_{opt}(\log(x-a),y)$.

It turns out I tell you the wrong $\alpha$ in comment. The optimal $\alpha$ should be around $16198.\color{red}{5}9255$.
Following is a plot of the prize pool vs. date.

• $y$-axis - the prize pool in unit of millions.
• $x$-axis - date number in unix universe (i.e time start at 1970/1/1 mid-night).
• the $\color{blue}{\text{blue}}$ curve is dataset from user.
• The $\color{red}{\text{red}}$ curve is the optimal fit using $\alpha = 16198.59255$: $$y = y_{opt}(x) = 2.067125632834611 \log(x-16198.59255) + 0.90491534594457$$
• The $\color{green}{\text{green}}$ curve is the fit using $\alpha = 16200$: $$y = y_{est}(x) = 1.417797964812813 \log(x-16200) + 2.558265172338731$$

On 2014/07/19 (unix date 16268), the two fit above gives following estimates of the prize pool (in unit of millions): $$y_{opt}(16268) \sim 9.66951606144537 \quad\text{ and }\quad y_{est}(16268) = 8.540674609249399$$

• Thank you so much for the elaborate reply. I had two questions to ask you however: I did plot a log graph over [1620, 16212] and did pretty much see a straight line, as you said. Why does this happen? Can you point me towards a book or paper that discusses this more in detail? Also, when computing $log(x - \alpha)$, how have you dealt with data points where $x \leq \alpha$. Once again, thank you for taking the time. May 24, 2014 at 7:18
• 1) For $x \gg y$, we have $\log(x+y) = \log(x) + \log(1+y/x) \sim \log(x) + \frac{y}{x}$. that's why you see a straight line. 2) $\alpha$ should be chosen to be smaller than all $x$. If for some reason you need to use a $\alpha$ bigger than a significant fraction of $x$, then $\log(x-\alpha)$ will not model the part of data for smaller $x$ properly. You need to split your dataset, use $\log(x-\alpha)$ (or something else) for larger $x$ and find another function for smaller $x$. At the end, find a way to merge the two fit to produce a single curve. May 24, 2014 at 7:34