Prove that the set A = {Positive (x,y) such that xy<=1} is nonconvex

I have taken two elements $$(x_1,y_1)$$ and $$(x_2,y_2)$$ in the set.

I want to check that $$(x_3,y_3) = \theta(x_1,y_1) + (1-\theta)(x_2,y_2) \le 1$$

I referenced Tom's answer on this question: Prove that the set $A$ $=$ {Positive $(x,y)$ such that $xy >= 1$} is convex.

So I need to check that $$\theta^2x_1y_1 + (1-\theta)^2x_2y_2 + \theta(1-\theta)[x_1y_2+x_2y_1] \le 1$$

So $$x_1y_1 \le 1$$, $$x_2y_2\le1$$

But $$[x_1y_2+x_2y_1] \ge 2\sqrt{x_1y_1x_2y_2} \le 2$$

I don't think I can use the same proof since the inequalities are in different directions.

To show that a set is nonconvex, you need the negation of the definition of convexity:

$$\exists(x_1,y_1),(x_2,y_2)\in A:\exists \theta \in (0,1):\theta(x_1,y_1)+(1-\theta)(x_2,y_2)\notin A$$

That is,

$$\exists(x_1,y_1),(x_2,y_2)\in A:\exists \theta \in (0,1):(\theta x_1+(1-\theta)x_2)(\theta y_1+(1-\theta)y_2)>1$$

and we see that $$(0.5, 2), (2,0.5)$$ with $$\theta = 0.5$$ does the trick:

$$(0.5\times 0.5+0.5\times 2)(0.5\times2+0.5\times 0.5)=1.25^2>1$$

Hence $$A$$ is nonconvex.

If the inequality in your question do hold, we would have proven that $$A$$ is convex.