Is $f(x,y) = x^2y + x y^2$ (quasi-) concave or convex? I should analyze whether the function
$$f(x,y) = x^2y + x y^2 \text{ where }  x,y > 0$$
is (quasi-) concave or convex.
Thus, as usual, I set up the Hessian as
$$ D^2f(x,y) = \left( \begin{array}{cc}
2y & 2x + 2y \\
2x + 2y & 2x \\
 \end{array} \right)  $$
which, given the constraints $x,y > 0$, is indefinite.
So I need the bordered Hessian:
$$
H= \left( \begin{array}{ccc}
0 & 2xy + y^2 & x^2 + 2xy \\
2xy + y^2 & 2y & 2x + 2y \\
x^2 + 2xy & 2x+2y & 2x \end{array} \right) $$
and check its determinant to find out about its properties. I end up with
$$ \det|H| = 2x(2x^3 + x^3 y - 2x^2 y + 4x^2 y^2 + 4 x y^3 + 2 y^4)$$
which, considering $x,y > 0$, looks quite positive to me, so $f$ would be quasiconcave.
But how can I prove that $\det|H| > 0$? Or is there any easier approach?
Thanks!
 A: The Hessian $D_2$ of your function $f$ is as you said, then:
$$
D_2 - \lambda I =
\begin{pmatrix}
2y - \lambda & 2x+2y
\\
2x+2y & 2x-\lambda
\end{pmatrix}.
$$
The determinant is:
$$
\det(D_2-\lambda I) = \lambda^2 - 2(x+y)\lambda - 4(x^2+y^2+xy) ,
$$
hence we have one positive and one negative eigenvalues, surely it is not positive semi-definite.
The bordered Hessian $H$ has determinant (I don't know what your expression in the question is, but it sure doesn't look right):
$$
\det (H) = 6 x^4 y + 12 x^3 y^2 + 12 x^2 y^3 + 6 x y^4 >0 \quad \forall x,y>0.
$$
Another way to do this is considering the level set 
$$F_a := \{(x,y): f(x,y)\ge t\}$$ is convex or not:

as you can tell by the level curve the level set is indeed convex, hence $f$ is quasi-concave. To see it mathematically: set
$$
x^2y + xy^2 -t =0 \implies y = -\frac{x}{2} + \sqrt{\frac{x^2}{4} + \frac{t}{x}},
$$
and
$$
y'' = \frac{6 t (t+x^3)}{x^{5/2} (4 t+x^3)^{3/2}} >0 \quad \forall t>0,
$$
therefore $F_a$ is convex and the quasi-concavity of $f$ follows.
