Awkward behavior of $x^2 \sin\frac{1}{x}$ at $x=0$?

Question

According to p. 107 of the book Advanced Calculus by Fitzpatrick,

Assuming that the periodicity and differentiability properties of the sine function are familiar, the following is an example of a differentiable function having a positive derivative at $x = 0$ but such that there is no neighborhood of $0$ on which it is monotonically increasing: $$f(x) = \begin{cases}x^2\sin 1/x & \text{if}\ x\neq 0 \\ 0 & \text{if}\ x = 0\end{cases}$$ The source of this counterintuitive behavior is that the derivative $f'$ is not continuous at $x = 0$.

There are two confusing things about it:

1. a. By the definition of derivative (at $x=0$), $$\lim_{x\to 0} \dfrac{f(x)-f(0)}{x-0} = \lim_{x\to 0} \dfrac{x^2 \sin \bigl(\frac{1}{x}\bigr)}{x} = 0,$$ since $\bigl\lvert\sin\bigl(\frac{1}{x}\bigr)\bigr\rvert \le 1$.

   b. By use of rules for the derivative of products and quotients, $$f'(0) = \biggl[ 2x \sin \biggl(\frac{1}{x}\biggr) + x^2 \biggl(-\frac{1}{x^2} \cos \biggl(\frac{1}{x}\biggr)\biggr) \biggr]_{x=0} $$ which is not defined since $\cos\bigl(\frac{1}{0}\bigr)$ is not defined.

   So why the text says that its derivative exists and its value is $>0$?

2. The function $f(x)$ is monotonically increasing because it is an odd function so for any neighborhood abound $x=0$, $f(x_1>0)>f(x_2<0)$. So why the text says otherwise?

Answer 1

1.- (a) is correct so $f'(0)=0$, but your conclusion on 1.- (b) is wrong. Notice that you are trying to compute the derivative of $f$ at $0$ by using the formula for $f$ on points different from zero. The formula $$ f'(x) = 2x \sin \biggl(\frac{1}{x}\biggr) + x^2 \biggl(-\frac{1}{x^2} \cos \biggl(\frac{1}{x}\biggr)\biggr) $$ holds for all $x\neq0$ but, as you know, this formula is undefined at $0$. Therefore, you can conlude that the derivative of $f$ is not continuous at $0$, since you cannot approximate $f'(0)$ by $f'(x)$ for $x$ near $0$. In other words, $$ f'(0) \neq \lim_{x\rightarrow 0} f'(x). $$

2.- In general, odd functions are not monotonically increasing. For example, $$ x^3 - x $$ is odd, but it is not monotonic. Indeed, $x^3-x$ is increasing on the intervals $(-\infty,-\tfrac{1}{\sqrt{3}})$ and $(\tfrac{1}{\sqrt{3}},\infty)$, but is decreasing on $(-\tfrac{1}{\sqrt{3}},\tfrac{1}{\sqrt{3}})$.

Back to $f(x) = x^{2}\sin(\tfrac{1}{x})$, you can tell that it is not increasing because it is positive on the intervals $(\tfrac{1}{2k\pi},\tfrac{1}{(2k-1)\pi})$ but negative on $(\tfrac{1}{(2k+1)\pi},\tfrac{1}{2k\pi})$ for every $k\neq0$. Since every neighborhood of $0$ contains infinitely many intervals of those types, we conclude that $f(x)$ has infinitely many changes of sign on any neighborhood of $0$. Thus, it is not monotonic near $0$.