prove or disprove $H$ is a subgroup If $H$ is a nonempty subset of a group $G,$ and if $a,b\in H,$ then $a^{-1}b^{-1} \in H,$ can we prove that $H$ is a subgroup of $G$? if not, how to disprove it?
 A: I will do two things in this answer. First, I will provide an example that shows that the condition in the question is not strong enough to guarantee that $H$ is a subgroup.
Then I will go on to show that it is actually very close to being strong enough, and classify those groups for which it is indeed strong enough.
First, for an example, take any group $G$ with an element $x$ of order $3$. Now, we see that the subset $\{ x\}$ satisfies the conditions, since it is non-empty, and the requirement is that $x^{-1}x^{-1}\in \{ x\}$ but since $x^{-1}x^{-1} = (x^2)^{-1} = (x^{-1})^{-1} = x$ this is indeed the case.
Now, it turns out that the part of the condition that is too weak is "non-empty". Note that if $H$ is to be a subgroup, then clearly the identity $e$ must be in $H$. I claim that if "non-empty" is replaced by "contains $e$" then $H$ will indeed be a subgroup.
To see this, we note first note that if $x\in H$ then also $x^{-1}e^{-1} = x^{-1}$ is in $H$, so $H$ contains inverses of all its elements. But now, if $x,y\in H$ then also $x^{-1}$ and $y^{-1}$ are in $H$, and thus so is $(x^{-1})^{-1}(y^{-1})^{-1} = xy$ so $H$ is a subgroup.
Now, I further claim that we can actually weaken this a bit, from "contains $e$" to "contains an element of finite order not divisible by $3$". And that this is in some sense the "weakest" it can be. More precisely, I will also show that if $x\in G$ has either infinite order or order divisible by $3$ then there is a subset $H$ of $G$ satisfying the conditions and containing $x$ but which is not a subgroup.
For this, we need a lemma, which may seem unrelated at first, but I will make it clear how it applies afterwards.
Lemma: Let $A\subseteq \mathbb{Z}$ be the smallest subset containing $1$ and closed under the operation $(n,m)\mapsto -(n+m)$.
Then $A = \{ n\in \mathbb{Z}\mid n\equiv 1\pmod{3}\}$.
Proof of Lemma: Denote by $B$ the set we claim that $A$ equals. First, we show that $A\subseteq B$, which by definition means we need to show that $1\in B$ and that for any $n,m\in B$ we have $-(n+m)\in B$. The first part is trivially true, and for the second, we note that if $n\equiv m\equiv 1\pmod{3}$ then $n+m\equiv 2\pmod{3}$ and thus $-(m+n)\equiv -2 \equiv 1\pmod{3}$ which was the second part.
To show that $B\subseteq A$ we proceed by induction (using the well-founded ordering "is closer to $0$").
Let $n\in B$. If $n > 0$ then we can write $-n = -2 + -(n-2)$ and since $-2\equiv -(n-2) \equiv 1\pmod{3}$ and both of these are in $A$ by the inductive hypothesis (since they are strictly closer to $0$ than $n$ is), this shows that $n = -(-2 + -(n-2))$ is in $A$.
Similarly, if $n < 0$ we can write $-n = 1 + (-n-1)$ and apply the same reasoning as above to see that $n = -(1 + (-n-1))$ is in $A$.
The way to apply this Lemma to the problem is to note that if $H$ is a subset satisfying the conditions and $x\in H$ then we get $x^n\in H$ for all $n\in A$, so by the Lemma, we get precisely $x^n$ for those $n$ that are $1$ mod $3$. On the other hand, if $x$ has either infinite order or order divisible by $3$, then this set of powers of $x$ does not contain $e$, but it satisfies the conditions, so we have shown the existence of a subset containing $x$ and satisfying the conditions, but which is not a subgroup.
On the other hand, if the order $k$ of $x$ is finite but not divisible by $3$ then either $k\equiv 1\pmod{3}$ or $-k\equiv 1\pmod{3}$ so either $k$ or $-k$ is in $A$, and if $H$ contains $x$ then it thus also contains $x^k = e$ or $x^{-k} = e$, and by the earlier remark, this ensures that $H$ is a subgroup.
All in all, we conclude that those groups for which the conditions are sufficient to guarantee $H$ to be a subgroup are precisely those where all elements have finite order not divisible by $3$ (so if $G$ is finite these are precisely those where $|G|$ is not divisible by $3$).
