$f(x)=x-x^2$ if $x$ is rational, $x+x^2$ if $x$ is irrational

$$f(x)= \left\{\begin{array}{ll} x-x^2 &\mbox{if x is rational,}\\ x+x^2 &\mbox{if x is irrational.} \end{array}\right.$$ Show that $f'(0)=1$ and yet there is no neighborhood $I$ of the point $0$ on which this function is monotonically increasing.

-
(i) Please don't post in the imperative (giving orders; "Show..." "Prove..." etc). (ii) If this is, by any chance, homework, please use the [homework] tag in addition to the subject tag. (iii) If you have a question, please ask the question. (iv) It would be most helpful if you specify the context in which you encountered this problem (for what course, reading what book), and where you are stuck or why you are having trouble; that way the answers you receive have a better chance of being aimed at the correct level and clearing up your doubts. – Arturo Magidin May 19 '11 at 2:38
Sorry. It's first time for me to enter this site. I didn't know how to ask the question. Next time, I would do better. – Jihyun Kim May 19 '11 at 7:26

To show that $f'(0)=1$, it suffices to prove that $\lim_{x\rightarrow 0}\frac{f(x)-x}x = 0$ (since $f(0) = 0$). Since $f(x)-x=\pm x^2$, it is always true that $|f(x)-x|=x^2$. Dividing by $x$ and making $x\rightarrow 0$ gives the answer.

As for the second part. Suppose such a neighborhood $I$ exists. Then let $x\in I$ be irrational. Since $\mathbb Q$ is dense in $\mathbb R$, there exists a sequence $r_n$ of rationals in $I$ such as $r_n\rightarrow x$ and $r_n > x$.

Since $r_n$ is rational, $f(r_n) = r-r_n^2$, on the other hand $f(x) = x+x^2$. $f$ is monotically increasing on $I$, so that $f(x)\leq f(r_n)$ for all $n$. In other words

$x+x^2\leq r_n-r_n^2$ for all $n$.

Passing to the limit, it is found $x+x^2\leq x-x^2$ which is absurd (if $x\neq 0$). Therefore $I$ does not exist.

-
I'm happy to see that your last sentence wasn't "Therefore $I$ do not exist." I would have been even happier if you had waited to see whether Jihyun chose to engage with Arturo Magidin's comments before posting your answer. We're trying to discourage questioners from certain modes of behavior, and to encourage others. – Gerry Myerson May 19 '11 at 4:18
@Gerry: On the plus side, it would seem Jihyun took at least some of that comment to heart, judging by his second posted question. – Arturo Magidin May 19 '11 at 5:10
@Arturo, yes. Score 1 for the good guys. – Gerry Myerson May 19 '11 at 6:17
It's the first time for me to use this site. Thank you for helping me. This is my homework assignment. I didn't know how I question. Is there anything rule I have to know and follow? – Jihyun Kim May 19 '11 at 7:24
@Jihuyn: Always use the [homework] tag for homework questions. Always say what you've tried and where you are stuck. Always give context. Never simply quote the assignment. – Arturo Magidin May 19 '11 at 16:04

$f'(x) := \lim_{\Delta x \to 0} \frac{f(x+\Delta x) - f(x)}{\Delta x}$

since f(0) = 0,

$f'(0) = \lim_{\Delta x \to 0} \frac{f(\Delta x)}{\Delta x}$

Using $\epsilon - \delta$ method : above equation is equal to (substitute k to $\Delta x$) "for every $\epsilon > 0$, there exists $\delta > 0$ that

$k \in (-\delta,\delta), k \neq 0 \implies \frac{f(k)}k \in (1-\epsilon,1+\epsilon)$."

If k is rational, then $f(k) = k - k^2$, and $\frac{f(k)}k = 1-k$. Therefore, if $\delta = \epsilon$, then $\frac{f(k)}k$ is between $1+\delta$ and $1-\delta$, which is $(1-\epsilon,1+\epsilon)$.

Similar method works if k is irrational ($\delta = \epsilon$)

$\therefore$ whether k is rational or not, for every $\epsilon>0$, there exists $\delta$.

$\therefore$, $f'(0) = 1$.

PS : some error can exists in $\epsilon - \delta$ method

-
I hope Jihyun gets something out of this. But I wish you had waited to see whether Jihyun might engage with Arturo Magidin's comments before posting your answer. – Gerry Myerson May 19 '11 at 4:20