0
$\begingroup$

I am trying to find a lower bound function for a set of data I have, and I am struggling with it. In the following graph the blue color is the set of data and the red color is my lower bound function.

inverse logarithm interpolation trial error

The data is bounded between $0$ and $1$, and looks a little bit like the inverse of the natural logarithm ($LN$), but with some initial noise, so my approach (by trial/error in Excel) was as follows (the red-colored line).

$f(n) = 0\quad,\quad n \in [4,9000]$

$f(n) = 0.98-\frac{1}{LN(n-9000)}\quad,\quad n\ge9000$

So i was able to approach to the data, but the initial segment of data $[4,9000]$ was avoided, because the initial growing of the data is smoother (and noisy) than the initial growing of the function $f(n)$ I prepared, so the lower bound is "broken" in that segment for some $n$.

For instance, in the worst case I could do exactly the same approach and create two or three more $f_i(s_i)$ functions to split the initial pending segment of data $n=[4..9000]$ in two, three or more segments $s_i$ with a similar lower bound function $f_i(s_i)$ based on the inverse logarithm but adapted to the specific segment $s_i$ of data.

Please I would appreciate very much if somebody could give me an idea about how could I make using only one function a closer lower bound, like the one I have drawn in black color.

Initially I would like to be able to get a closer function by using the natural logarithm or a similar approach, not using a "pure" polynomial interpolation.

In other words, what I am looking for is a "parametrized" natural logarithm-based (or "similar to natural logarithm-based") lower bound function more than a "pure" parametrized polynomial interpolation, any solution is welcomed but I would appreciate very much a natural logarithm interpolation if possible. Thank you!

$\endgroup$
1
$\begingroup$

To my eye, the noise is about $\pm (10-20)\%$ of $(1-$ the curve). I would be tempted to smooth the data, fit a curve, then reduce it by a "three sigma" amount to produce a curve that is below "almost all" the points. It seems to me that would be a better representation of the data. Is that acceptable? The scare quotes are meant to indicate that a few points would fall below the curve you quote as a lower bound, but that may or may not be acceptable for your application. If you need a curve that is truly a lower bound, you will be farther from the "by eye" fit.

$\endgroup$
  • $\begingroup$ thank you very much, I will check your proposal and come back again, but sorry, to be sure, by "three sigma", do you mean this? Thank you! en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule $\endgroup$ – iadvd Mar 20 '15 at 4:15
  • $\begingroup$ I was thinking very informally, but the idea is the same. If you have a model of the noise, say Gaussian with $\sigma=0.1(1-data)$ you could make a "lower bound" by subtracting three times $\sigma$ from your fit. That doesn't guarantee that there won't be a few points lower. There may even be more than one in 2000 lower, because most real distributions have much wider tails than the normal distribution. You need to decide what you want. From your data, $-10$ is a lower bound, but not a useful one. Is it more useful to have a function that is truly a lower bound, $\endgroup$ – Ross Millikan Mar 20 '15 at 4:26
  • $\begingroup$ or one that is below "almost all" the points, but closer to most of them? That is a hard question, and depends on what you are going to do with the function. If only one point in a million is below your "lower bound", but there is one chance in ten thousand that your model is badly wrong (and many points might be below your bound), should you worry about that point? It is hard $\endgroup$ – Ross Millikan Mar 20 '15 at 4:30
  • $\begingroup$ thanks again for taking time for this! the data should be strictly bounded, all the points must be over it. Initially I would like to find a "standard" function able to follow the growth of the data from the beginning, maybe the natural logarithm is not the best one. E.g. maybe another family of functions is able to get closer, and then having that function as the base, I would attack the noise problem. I will keep the question open for a while just in case some ideas regarding that possibility arrive. If not then I would try a solution in the way you mention. :) $\endgroup$ – iadvd Mar 20 '15 at 5:18
  • $\begingroup$ I was able to find a better bound, but finally it was not possible to use only one single function for the whole range of values. I have added the closest solution so far to the answers. :) $\endgroup$ – iadvd Mar 23 '15 at 7:46
0
$\begingroup$

Well, finally I was able to find a better mix as follows:

Final lower bound

The initial segments are just simple polynomial approximations, and finally I found a better option for the last segment, including a combination of the inverse of the natural logarithm and $\frac{1}{x}$. This could give ideas to more people about how to attack this kind of problems, so I am adding it to the solutions.

$f(n) = 0,\quad n\lt1918$

$f(n) = (\frac{n^2-n}{(3300^2-3300)}*0.63)\quad,\quad n\lt3300$

$f(n) = 0.63\quad,\quad n\lt6000$

$f(n) = 0.71\quad,\quad n\lt9000$

$f(n) = \frac{4.95-\frac{1}{n}+\frac{LN(n)}{2.7}}{10}\quad,\quad n\ge9000$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.