# Why is error term in the definition of derivative for $\mathbb{R}^n \to \mathbb{R}^m$ of order $o(h)$?

The following intuition is usually supplied for a definition of derivative. We would like to approximate function $$f(x)$$ near some point $$x_0$$ with a linear map and we would like to show that as we get closer to $$x_0$$, this approximation becomes good (for certain notion of good).

The following equation is true in general (by definition of error term $$\varepsilon(h)$$).

$$f(x_0+h) - f(x_0) = L(h) + \varepsilon(h)$$

Now, as $$L(h)$$ and $$\varepsilon$$ are vectors of $$\mathbb{R}^m$$, to compare them it is reasonable to use magnitude. So, "intuitively", we would like magnitude of relative error to become small as we approach $$x_0$$. In other words, we would like:

$$\lim \limits_{h \to 0} R(h) = \lim \limits_{h \to 0} \left( \frac{|\varepsilon(h)|}{|L(h)|} \right) = 0.$$

Question 1: This expression does not make sense if $$L(h) = 0$$ which could certainly be a derivative for some function. How can in that case we say that function $$0$$ approximates function close to $$x_0$$ if relative error is undefined? Do we use some different criterion? Do we ignore this case?

Now, if we assume limit is defined and exists, we can use that $$|L(h)| \leq M|h|$$ as it is a linear map $$\mathbb{R}^n \to \mathbb{R}^m$$. Then, we get the following.

$$0 = \lim \limits_{h \to 0} R(h) \geq \frac{1}{M} \lim \limits_{h \to 0} \left( \frac{|\varepsilon(h)|}{|h|} \right)$$

From here it seems to be if we assume (?) limit exists and assume it is zero by out intuition on what good approximation is, we can show that it necessarily means that error is of order $$o(h)$$.

Question 2: Is this reasoning valid?

Question 3: The other direction -- if error term is $$o(h)$$ -- does not seem to imply that relative error tends to $$0$$. So, could it be that derivative is defined, but its relative error compared to non linear term does not vanish? How to interpret this with mindset of derivative being the best linear approximation?

• We want the error to go to zero faster than $|h|$, i.e we want $\lim_{h\to 0}\dfrac{|\epsilon(h)|}{|h|} = 0$ (that's the definition of differentiability), not $\lim_{h\to 0}\dfrac{|\epsilon(h)|}{|L(h)|}=0$, because like you said this limit need not even make sense Aug 10, 2020 at 21:48
• @peek-a-boo yes. My question is why (except it is defined this way, of course). What is intuition behind this? I tried to give intuition but there are some things I don't like with it. What do you think about my reasoning and the doubts I have? Thanks! Aug 10, 2020 at 21:49

By definition a function is differentiable when the following holds

$$f(x_0+h) - f(x_0) = L(h) + \varepsilon(h)$$

with

• $$L(h)=\nabla f(x_0)\cdot h$$
• $$\varepsilon(h)=o(|h|) \implies \lim \limits_{h \to 0} \frac{|\varepsilon(h)|}{|h|} = 0$$

form this definition we can prove that when a function is differentiable it is also continuous and all directional derivatives exist.

• Would you say that then the intuition for derivative is not that it is a good linear approximation but rather that it is a linear function from which we can prove continuity of original function $f$ and that directional derivatives exist? In this case, could it be that we choose different order for error term and still get these theorems? Aug 10, 2020 at 21:59
• Frechet differentiable, yes, but not necessarily so for Gateaux differentiable, in $\ge 2$ dimensions. Aug 10, 2020 at 22:00
• @DanielsKrimans For function $\mathbb R \to \mathbb R$ derivative and differentiabilty are equivalent concepts, but for function of several variables things are more complicated and less intuitive. For example, the existence of all directional derivatives at a point does not imply continuity. But the intuition is the same, differentiability means that locally the function can be approximated by a linear operator, which is the gradient.
– user
Aug 10, 2020 at 22:07
• @kimchilover I'm not aware about Gateaux differentiability.
– user
Aug 10, 2020 at 22:08
• @user See en.wikipedia.org/wiki/… . The difference comes down to $\lim{t\to0}\epsilon(tv)/t=0$ holding pointwise in the vector $v$ or uniformly on bounded sets of $v$ (or, if it makes a difference, uniformly in $v$ on compact sets). Aug 10, 2020 at 22:17

Think in the simple case for maps $$\mathbb{R} \to \mathbb{R}$$. Suppose we're given a differentiable function $$f$$ and some point $$x_0$$. Many linear maps will "approximate" $$f$$ at $$x_0$$, in the sense that the difference between that linear map and $$f$$ tends to $$0$$ as $$x \to x_0$$. Indeed, any linear map of the form $$\alpha(x-x_0) + f(x_0)$$ for $$\alpha \in \mathbb{R}$$ works. But the best linear approximation is the one where the error tends to $$0$$ much faster than we would expect it to, namely it tends to $$0$$ like $$o(x-x_0)$$. This corresponds to a tangent line geometrically.

• Thanks! It is not clear to me why you say this is the best linear approximation -- by which metric? As I gave already an argument that it could be that it goes like $o(x-x_0)$ but relative error of linear term with respect to error term does not go to zero. Aug 10, 2020 at 22:07