# How to determine convexity of the set

Let $$S=\left\{(x_1,x_2)\in \mathbb{R}^2: \sqrt[4]{2x_1^4+2x_1^2x_2+x_2^2}\leq 5 \right\}\cap\left\{(x_1,x_2)\in \mathbb{R}^2: \cos(x_1)+3x_1^2+x_2\leq 5 \right\}$$

I want to determine, whether S is convex. Set $$\left\{(x_1,x_2)\in \mathbb{R}^2: \cos(x_1)+3x_1^2+x_2\leq 5 \right\}$$ is convex, by considering Hessian of $$\cos(x_1)+3x_1^2+x_2$$ and using the fact that the level set $$\left\{x: f(x)\leq \alpha \right\}$$ is convex for all $$\alpha \in \mathbb{R}$$ if $$f$$ is convex. Also, intersection of two convex sets is convex, this would mean that $$S$$ is convex provided $$\left\{(x_1,x_2)\in \mathbb{R}^2: \sqrt[4]{2x_1^4+2x_1^2x_2+x_2^2}\leq 5 \right\}$$ is a convex set. We can rewrite condition defining this set as: $$f(x_1,x_2)=x_1^4+(x_1^2+x_2)^2\leq 5^4$$ but $$f(x_1,x_2)$$ is not convex as $$2x_1^2x_2$$ is not convex. Would this imply that $$S$$ is not convex, since there are points $$(x_1,x_2)\in S$$ for example $$(0,-1)\in S$$ for which Hessian of $$f(x_1,x_2)$$ is not positive semi-definite.

If you plot the set, it's clear that it's nonconvex:

To explicitly prove that it's nonconvex, note that the points $(-3,-30), (3,-30)$ are contained in the set but their midpoint $(0,-30)$ is not. That is:

\left.\begin{aligned}\cos(\pm 3) + 3 (\pm3)^2 - 30 &\le 5 \\ (\pm3)^4 + ((\pm3)^2 - 30)^2 &\le 5^4\end{aligned}\right\} \Rightarrow (\pm 3,-30) \in S\\ (0)^4 + ((0)^2 - 30)^2 > 5^4 \Rightarrow (0,-30) \notin S

I should point out that while your calculation does suggest that $S$ is nonconvex, it's not quite a proof for two reasons. First, while sublevel sets of convex functions are convex sets, the sublevel sets of nonconvex functions are not necessarily nonconvex sets. Second, if a set is an intersection such as $S=A \cap B$, then $B$ being nonconvex does not imply that $S$ is nonconvex, since we could have $A$ convex and $A \subset B$. As it turns out, neither of these issues matter in this particular case however.