Find the extrema of given function $\mathbb{R}^2\to\mathbb{R}$? Let's consider the function $$f(x,y)=\begin{cases}\arctan(xy)-x^2,& \text{ where }y\leq x\\xy-\arctan(x^2),&\text{ where }y>x\end{cases}$$


*

*Is this function continuous on $\mathbb R^2$?

*What are all absolute extrema on the set $M=\{(x,y)\in\mathbb R^2:0\leq y\leq x\leq 1\}$?


As I see it the components of the function are each continuous. The problem however lies in the axis where $x=y$ where it is not continuous. To show that I took two sequences $(x_n,y_n)=(1,1+\frac{1}{n})$ and $(\tilde x_n,\tilde y_n)=(1,1-\frac{1}{n})$ that converge into $(1,1)$ but $f(x_n,y_n)\to \frac{\pi}{4}-1$ and $f(\tilde x_n,\tilde y_n)\to 1-\frac{\pi}{4}$ and I could do the same for every other point on said axis.
Now however I am a bit confused when it comes ot the extrema of $f$. We have $$\nabla f=\left(\frac{y}{x^2+y^2-1}-2x,\frac{x}{x^2y^2+1}\right)^T$$ so there is only one absolute extreme value that can be taken at $(0,0)$. But can I now just take the hessian to determine what kind of point $f$ has? I mean, it is not contiuous in that point so technically I can't compute its derivative there but since $f(0,0)=0$ and in every neighbourhood of $(0,0)$ I find both positive and negative values for $f(x,y)$ it is kind of obvious that it must be a saddle point.
And what is with the rest? Since $\nabla (f)$ has no zeros in $M$ does this mean that all the extreme values are taken on $\partial M$? If so I would look at the borders separately:


*

*$f(x,0)=-x^2$ for all $(x,0)\in M$ so $f$ takes values in between $0$ and $-1$ here.

*$f(x,x)=\arctan(x^2)-x^2$ for all $(x,x)\in M$ so $f$ takes values in between $0$ and $\frac{\pi}{4}-1$.

*$f(1,y)=\arctan(y)-1$ for all $(1,y)\in M$ so $f$ takes values in between $\frac{\pi}{4}-1$ and $-1$.


All in all I can conclude now that $f$ has the global maximum $0$ which is taken at $(0,0)$ and the global minimum $-1$ which is taken at $(1,0)$. Is that correct?
 A: A few minor points. There are some computational errors (perhaps typos?): $f(x_n,y_n)\to 1-\frac \pi 4$, $f(\tilde x_n,\tilde y_n)\to\frac \pi 4-1$, and the partial derivatives for the first branch are
$$f_x=\frac{y}{1+x^2y^2}-2x\quad\quad f_y=\frac{x}{1+x^2y^2} $$ 
Also, the proof you gave for discontinuity works for points $(a,a)$ where $a\ne 0$. That's enough to answer the first question but it looks like the function is continuous at $(0,0)$.
As for the second question, the function restricted to $M$ is continuous, and $M$ is a closed and bounded set. By the extreme value theorem, it has extrema on that set, and they occur either on the boundary or in the interior of $M$. If they were in the interior (where the partial derivatives exist), it would follow that $f_x=0$ and $f_y=0$ which leads us to $(x,y)=(0,0)$ and that's not in the interior. So, you can conclude the extrema occur on the boundary, and you have to check what happens there. You've already done this part. Notice that we didn't need the Hessian here.
