# Estimating a function given a noisy sequence of its output

I am new to this forum. Please forgive me if this question is elementary, but I am somewhat lost and could use a little guidance.

Suppose I have an unknown function $f(i)=x_i$. I have a sequence of observations $y_1,y_2,\ldots,y_n$ of sequence $x_1,x_2,\ldots,x_n$, which is the result of applying this function on $\{1,2,\ldots,n\}$. The observations have a Gaussian error "skirts" around them such that:

$$p(y|x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y-x)^2}{2\sigma^2}}$$

Now, if the function is linear and there is no autocorrelation, I think that I can use the usual least squares method that I learned in the undergraduate econometrics to obtain the coefficients, and, (I assume, since I've never have actually done that, but it seems reasonable) knowing $\sigma^2$ would get me a nice expression of some kind for the errors (my guess would be that they'd be Gaussian with mean zero and variance $\sigma^2$ or something like that).

If I am not too off-base in the previous paragraph, what do I do if $f(i)=x_i$ is autocorrelated, such that $x_i$ depends on $i-1,i-2,\ldots, i-m$ for some $m<i$? I would appreciate if someone could point me to the reading that would gently shed light on this topic for me? I am reasonably proficient in calculus (undergraduate vector calculus course), and have some background in linear algebra (for example, I know that the least squares method works because errors are set to be orthogonal to the data vectors), but most of my math knowledge is limited to the requirements of my economics major. Seems to me that my question would be pretty well-studied though, and an intuitive explanation available...

Thank you!

-
There is a site for stats questions. If you think you will get better answers there, you could ask the moderators to migrate your question to the stats site. – Gerry Myerson Feb 10 '12 at 6:16
I was looking for a more "fundamental intuitive math" answer, so to say. But, yeah, perhaps stats site might be more helpful. – HellRazor Feb 10 '12 at 6:20
Would you happen to have an idea on the probability distribution your noise is following? – J. M. Feb 10 '12 at 6:44
The observations are distributed as Gaussians with mean being the "true" value, and known variance. For the autocorrelated version, I guess, for simplicity, one can assume that errors are independent from observation to observation, with the same variance. – HellRazor Feb 10 '12 at 6:51
Have a look at autoregressive models. These assume that the current same is a linear combination of previous $M$ samples with some noise added, and might be what you were looking for. – Kartik Audhkhasi Aug 24 '12 at 14:33

Making no assumptions on $f(i)=x_i$, it will be impossible to estimate its behavior with by observing its noisy output.

You could have for example:

$f(i_n, i_{n-1})=g(i_n-i_{n-1})$

i.e. $f$ could depend only on the difference between subsequent inputs. In this case, if the input sequence is the one described, you are clearly seeing very little about the possible outputs of your function $f$.

If you can suppose $f$ only depends on cuttent input $i$, you will anyways need to assume some properties $f$ e.g. that i can be piecewise approximated by a class of functions. If this is your case, I would start looking into Polinomial Approximation or Splines.

If no assumption can be made about $f$ but that it is memoryless, then with a sigle input sequence as the one you describe, your best estimate for the cofficients $x$ is the actual noisy output $y$.

-