# Is $f(x,y) = x^2y + x y^2$ (quasi-) concave or convex?

I should analyze whether the function

$$f(x,y) = x^2y + x y^2 \text{ where } x,y > 0$$

is (quasi-) concave or convex.

Thus, as usual, I set up the Hessian as

$$D^2f(x,y) = \left( \begin{array}{cc} 2y & 2x + 2y \\ 2x + 2y & 2x \\ \end{array} \right)$$

which, given the constraints $x,y > 0$, is indefinite.

So I need the bordered Hessian:

$$H= \left( \begin{array}{ccc} 0 & 2xy + y^2 & x^2 + 2xy \\ 2xy + y^2 & 2y & 2x + 2y \\ x^2 + 2xy & 2x+2y & 2x \end{array} \right)$$

and check its determinant to find out about its properties. I end up with

$$\det|H| = 2x(2x^3 + x^3 y - 2x^2 y + 4x^2 y^2 + 4 x y^3 + 2 y^4)$$

which, considering $x,y > 0$, looks quite positive to me, so $f$ would be quasiconcave. But how can I prove that $\det|H| > 0$? Or is there any easier approach?

Thanks!

The Hessian $D_2$ of your function $f$ is as you said, then: $$D_2 - \lambda I = \begin{pmatrix} 2y - \lambda & 2x+2y \\ 2x+2y & 2x-\lambda \end{pmatrix}.$$ The determinant is: $$\det(D_2-\lambda I) = \lambda^2 - 2(x+y)\lambda - 4(x^2+y^2+xy) ,$$ hence we have one positive and one negative eigenvalues, surely it is not positive semi-definite.
The bordered Hessian $H$ has determinant (I don't know what your expression in the question is, but it sure doesn't look right): $$\det (H) = 6 x^4 y + 12 x^3 y^2 + 12 x^2 y^3 + 6 x y^4 >0 \quad \forall x,y>0.$$
$$F_a := \{(x,y): f(x,y)\ge t\}$$ is convex or not:
as you can tell by the level curve the level set is indeed convex, hence $f$ is quasi-concave. To see it mathematically: set $$x^2y + xy^2 -t =0 \implies y = -\frac{x}{2} + \sqrt{\frac{x^2}{4} + \frac{t}{x}},$$ and $$y'' = \frac{6 t (t+x^3)}{x^{5/2} (4 t+x^3)^{3/2}} >0 \quad \forall t>0,$$ therefore $F_a$ is convex and the quasi-concavity of $f$ follows.