How to check convexity? How can I know the function $$f(x,y)=\frac{y^2}{xy+1}$$ with $x>0$,$y>0$ is convex or not?
 A: The book "Convex Optimization" by Boyd, available free online here, describes methods to check.
The standard definition is if f(θx + (1 − θ)y) ≤ θf(x) + (1 − θ)f(y) for 0≤θ≤1 and the domain of x,y is also convex.
So if you could prove that for your function, you would know it's convex. 
The Hessian being positive semi-definite, as mentioned in comments, would also show that the function is convex.
See page 67 of the book for more.
A: Consider $y=x$ then we have $\displaystyle g(x)=\frac{x^2}{x^2+1}=1-\frac 1{x^2+1}$
The second derivative of this is $g''(x)=\frac{2-6x^2}{(1+x^2)^3}$
and will change sign around $x=\frac 1{\sqrt{3}}$ so that $g$ is convex in $(0,\frac 1{\sqrt{3}})$ and concave in $(\frac 1{\sqrt{3}},\infty)$.
Your function is clearly not convex nor concave on $(\mathbb{R^{+*}})^2$ but you could search more restricted sets if needed...
Here is a picture (from below) of your function (convex near $y=0$ and concave when $y$ becomes larger at least in the x=y direction, in the x=-y direction it looks convex...) : 

A: The Hessian of $\frac{y^2}{1+xy}$ is
$$
H
=
\frac1{(1+xy)^3}\begin{bmatrix}
2y^4&-y^2(3+xy)\\[12pt]
-y^2(3+xy)&2
\end{bmatrix}
$$
and
$$
\begin{bmatrix}
u&v
\end{bmatrix}
H
\begin{bmatrix}
u\\v
\end{bmatrix}
=\frac2{(1+xy)^3}(v-uy^2)^2-\frac1{(1+xy)^2}uvy^2
$$
Setting $\begin{bmatrix}u&v\end{bmatrix}=\begin{bmatrix}1&y^2\end{bmatrix}$ gives
$$
\begin{bmatrix}
1&y^2
\end{bmatrix}
H
\begin{bmatrix}
1\\y^2
\end{bmatrix}
=-\frac{y^4}{(1+xy)^2}
$$
so $\frac{y^2}{1+xy}$ is not convex as long as $y\ne0$.
