How to determine convexity of the set

Let $S=\left\{(x_1,x_2)\in \mathbb{R}^2: \sqrt{2x_1^4+2x_1^2x_2+x_2^2}\leq 5 \right\}\cap\left\{(x_1,x_2)\in \mathbb{R}^2: \cos(x_1)+3x_1^2+x_2\leq 5 \right\}$

I want to determine, whether S is convex. Set $\left\{(x_1,x_2)\in \mathbb{R}^2: \cos(x_1)+3x_1^2+x_2\leq 5 \right\}$ is convex, by considering Hessian of $cos(x_1)+3x_1^2+x_2$ and using the fact that the level set $\left\{x: f(x)\leq \alpha \right\}$ is convex for all $\alpha \in \mathbb{R}$ if $f$ is convex. Also, intersection of two convex sets is convex, this would mean that S is convex provided $\left\{(x_1,x_2)\in \mathbb{R}^2: \sqrt{2x_1^4+2x_1^2x_2+x_2^2}\leq 5 \right\}$ is a convex set. We can rewrite condition defining this set as: $f(x_1,x_2)=x_1^4+(x_1^2+x_2)^2\leq 5^4$ but $f(x_1,x_2)$ is not convex as $2x_1^2x_2$ is not convex. Would this imply that $S$ is not convex, since there are points $(x_1,x_2)\in S$ for example $(0,-1)\in S$ for which Hessian of $f(x_1,x_2)$ is not positive semi-definite.

If you plot the set, it's clear that it's nonconvex: To explicitly prove that it's nonconvex, note that the points $(-3,-30), (3,-30)$ are contained in the set but their midpoint $(0,-30)$ is not. That is:
\left.\begin{aligned}\cos(\pm 3) + 3 (\pm3)^2 - 30 &\le 5 \\ (\pm3)^4 + ((\pm3)^2 - 30)^2 &\le 5^4\end{aligned}\right\} \Rightarrow (\pm 3,-30) \in S\\ (0)^4 + ((0)^2 - 30)^2 > 5^4 \Rightarrow (0,-30) \notin S
I should point out that while your calculation does suggest that $S$ is nonconvex, it's not quite a proof for two reasons. First, while sublevel sets of convex functions are convex sets, the sublevel sets of nonconvex functions are not necessarily nonconvex sets. Second, if a set is an intersection such as $S=A \cap B$, then $B$ being nonconvex does not imply that $S$ is nonconvex, since we could have $A$ convex and $A \subset B$. As it turns out, neither of these issues matter in this particular case however.