How to study the differentiability of this function? Given this function $f:\mathbb R^2\to\mathbb R$, can you tell if it's differentiable on $(0,0)$?
$$f(x,y)=\begin{cases}\cfrac{x^ny^m}{x^2-xy+y^2}&\text{for }(x,y)\ne(0,0)\\0&\text{for }(x,y) =(0,0)\end{cases}$$
$m,n \in \mathbb N^*$.
I want to study 2 cases of $m$ and $n$, to make a generalization of the problem. When:
1) $n=m=1$, $f$ is not continuons $\Rightarrow$ $f$ is not differentiable
2) $m+n>2 \iff m\gt 1$ or $n>1 \iff m\ge2$ or $n\ge2$, $f$ is continuons
How should I study the differentiability in the second case $(m+n>2)$?
 A: Allow me to start by making general comments.
First it's always a good idea to try to figure out a priori whether the function is differentiable or not, because proving one way or the other are two quite different jobs, in two words:


*

*If you want to show the function is differentiable, you don't have that many tools at your disposal. Either you can use some rules such as "the product of differentiable functions is differentiable", or you have to do the job "by hand" coming back to the definition. In this latter case you probably first need to figure what the "good candidate" for the differential is.

*If you want to show the function is not differentiable, you probably have to show why it being differentiable would yield a contradiction. There might be various ways to go about that (for example, easiest is to show that the function is not even continuous, like you suggest for $m = n = 1$).


Personally I find it useful to remember the following fact : when a function $f$ is differentiable at $m$, its differential tells you how velocities at $m$ of curves through $m$ are transformed. In other words for any (say ${\cal C}^1$) curve $c$ such that $c(0) = m$, we have $(f \circ c)'(0) = df_{m} (c'(0))$. It's a good way to think of what a differential is anyway.
Let's come back to your example (with $m+n > 2$) and let's assume for a moment that $f$ is indeed differentiable at $(0,0)$.  It's probably a good idea to try the curves $c_1(t) = (t, 0)$, $c_2(t) = (0,t)$, $c_3(t) = (t,t)$, see what we get.
The first one tells you that $(f\circ c_1)'(0) = 0 = df_{(0,0)}(c_1'(0)) = df_{(0,0)}((1,0))$ in other words $\frac{\partial f}{\partial x}(0,0) = 0$. 
The second one tells you similarly that $\frac{\partial f}{\partial y}(0,0) = 0$.
At this point we already know that if $f$ is indeed differentiable at $(0,0)$, then its differential $df_{(0,0)}$ must be the zero linear map. So now either we want to find a contradiction and conclude that $f$ is not differentiable, or we want to show that $df_{(0,0)} = 0$ holds indeed.
Let's look at what the third curve tells us: since $f(t,t) = t^{m+n -2}$, we find $df_{(0,0)}((1,1)) = 0$ if $m + n > 3$ and $df_{(0,0)}((1,1)) = 1$ if $m+n = 3$.
So if $m + n = 3$, we have a contradiction: this shows that $f$ cannot be differentiable at $(0,0)$.
Let's assume now that $m + n > 3$, so we have no contradiction so far. Usually at this point it's reasonable to think that the function is likely differentiable, and we have the candidate for its differential, so let's try to prove it by coming back to the definition. Sometimes though, it turns out that the function is actually not differentiable and maybe you need to look for a more clever curve, such as $(t^2, t^3)$.
Anyway, let's prove directly that $f$ is differentiable at $(0,0)$ and $df_{(0,0)} = 0$. Everything I've written above is unnecessary for what follows (say it's a preliminary investigation). By definition, we need to show that 
$$ f(0+h_1, 0+h_2) = f(0,0) + df_{(0,0)}(h_1, h_2) + o(\Vert (h_1, h_2) \Vert)$$
in other words 
$$\lim_{\Vert (h_1, h_2) \Vert \to 0} \frac{ \left| f(h_1, h_2) \right|}{\Vert (h_1, h_2) \Vert}= 0 ~.$$
At least now you know precisely what you need to show. I could let you finish the job but while I'm at it...
Let's take the norm $\Vert(h_1, h_2) \Vert = \sqrt{h_1^2 + h_2^2}$ (any norm will do). Let's look at the denominator of $\left|f(h_1, h_2)\right|$, we want to find a good lower bound:
$$ \left|h_1 ^2 - h_1 h_2 + h_2^2 \right| \geqslant \left| h_1^2 + h_2^2 \right| - \left|h_1 h_2\right|$$
Recall that for any real numbers $x$ and $y$, we always have $\left|xy\right| \leqslant \frac{1}{2}(x^2 + y^2)$, so we get
$$ \left|h_1 ^2 - h_1 h_2 + h_2^2 \right| \geqslant \frac{1}{2}\Vert| (h_1,h_2) \Vert^2 ~.$$
And now the numerator: note that either $n \geqslant 3$, or $m \geqslant 3$, or $n \geqslant 2$ and $m \geqslant 2$. Let's treat the first case for example: $n \geqslant 3$. So $\left|h_1^n h_2^m\right| \leqslant \left|h_1^2\right|\, \left|h_1 h_2 \right|$. Since $\left|h_1^2\right| \leqslant \Vert| (h_1,h_2) \Vert^2$ and $\left|h_1 h_2\right| \leqslant \frac{1}{2}\Vert| (h_1,h_2) \Vert^2$, we get $\left|h_1^n h_2^m\right| \leqslant \frac{1}{2}\Vert| (h_1,h_2) \Vert^4$. I'll let you treat the two other cases to find similar inequalities (find the same inequality for second case, find $\frac{1}{4}$ instead of $\frac{1}{2}$ for the third case).
Putting things together, we finally get $$\left|f(h_1, h_2)\right| \leqslant \Vert| (h_1,h_2) \Vert^2$$
(or $\left|f(h_1, h_2)\right| \leqslant \frac{1}{2}\Vert| (h_1,h_2) \Vert^2$ in the case $n \geqslant 2$ and $m \geqslant 2$, whatever) and the conclusion follows.
Phew!
