# Existence and uniqueness of least square fits

I just noticed that my pythons scipy library gave me another solution to an exponential curve fit than my pocket calculator.

If you plot both solutions both match the data (visually) very well. This lead me to the question of what can be said about existence and uniqueness of least square fits.

So consider a set of finite points $$(x_i,y_i), i \in \{1,\dots,n\}, n \in \mathbb{N}$$. And a function $$f\colon D \supset \mathbb{R} \to \mathbb{R}$$ which depends on parameters $$a_j, j \in \mathbb\{1,\dots,k\}, k \in \mathbb{N}$$.

Now one wants to find the parameters $$(a_j)$$ such that $$\sum_i (f(x_i) -y_i)^2$$ is minimal.

What results are there about existence and uniqueness of $$f$$?

I am interested in particular in the case that $$f$$ is an elementary function such as

• polynomial function of the form $$f(x) = \sum_l a_l x^l$$
• exponential function of the form $$f(x) = a_0 \cdot a_1^x$$
• something reciprocial like $$f(x) = a_0 + a_1/x^m$$
• logarithmic function like $$f(x) = a_0 + a_1 \ln(x)$$

If in some of those cases the solution doesn't exist or is not unique, I am looking for simple counterexamples. I am also interested in good references (which includes the proofs) about this special topic.

The dependence of $$f$$ on $$x$$ is not very important, it's the dependence on the parameters $$a_j$$ for each fixed $$x$$ that counts. In particular, in each of your cases except the second $$f$$ is linear in the $$a_j$$, and then $$\sum_i (f(x_i)-y_i)^2$$ is convex in the vector $$\bf a$$. Moreover, unless the $$f(x_i)$$ are linearly dependent, it is strictly convex, which implies that a minimum is unique, and $$\sum_i (f(x_i)-y_i)^2 \to \infty$$ as $$\|\bf a\| \to \infty$$, which implies that the minimum exists.
• Just to be clear, I did not claim the minimum is unique in the exponential case. The residuals $(f(x_i) - y_i)^2$ are not in general convex in that case. – Robert Israel Mar 13 '19 at 12:30
• For the exponential case, note that if all $x_i$ are even integers, $(a_0, a_1)$ and $(a_0, -a_1)$ give the same results. You might not like this example because $f(x)$ is not real when $a_1 < 0$ and $x$ is not an integer. But I suspect examples of nonuniqueness can also be found where the $a_1$ values are positive. – Robert Israel Mar 13 '19 at 12:35