How much of a group $G$ is determined by the category of $G$-sets? Suppose $G$ and $H$ are groups and we have an equivalence of categories between $G\textrm{-}\mathbf{Set}$ and $H\textrm{-}\mathbf{Set}$. (One can think of this as a form of "nonlinear Morita equivalence".) What can be said about $G$ and $H$? I suspect that $G$ and $H$ have to be isomorphic, but I can't prove it. If this is too hard in general, I'm also interested in the case that $G$ and $H$ are assumed to be finite.
Here are some things I've tried:
Since $G$-modules are the abelian group objects in $G\textrm{-}\mathbf{Set}$, we get a Morita equivalence between $\Bbb{Z}[G]$ and $\Bbb{Z}[H]$. So if we tensor with any field $k$, we get a Morita equivalence between $k[G]$ and $k[H]$.
Assume for a moment that $G$ and $H$ are finite. If we take $k = \Bbb{C}$, then this means that $G$ and $H$ have the same number of irreducible representations, so the same number of conjugacy classes. If we take $k=\Bbb{R}, \Bbb{Q}, \overline{\Bbb{F}_p}$, we also get the same number of real, rational and $p$-regular conjugacy classes.
While this approach gives some common properties between $G$ and $H$, it's not possible to conclude that $G$ and $H$ are isomorphic, because there are known examples of non-isomorphic finite groups with isomorphic integral group algebras. (See here) This means that we have to use some of the "nonlinear" information from the category $G\textrm{-}\mathbf{Set}$.
Another thing I considered is that the automorphism group of the forgetful functor $G\textrm{-}\mathbf{Set} \to \mathbf{Set}$ is isomorphic to $G$, by a simple Yoneda-argument, so if we could somehow reconstruct the forgetful functor just from the category $G\textrm{-}\mathbf{Set}$, this would show that $G$ and $H$ must be isomorphic. I haven't been able to do this, but I reconstructed some other functors: The terminal object in $G\textrm{-}\mathbf{Set}$ is a one-point set with a trivial action, denote this $G$-set by $\{*\}$, we have a natural bijection $\operatorname{Hom}_{G\textrm{-}\mathbf{Set}}(\{*\},X) \cong X^G$, where $X^G$ denotes the set of fixed points under the action of $G$.
So we can reconstruct the fixed point functor $G\textrm{-}\mathbf{Set} \to \mathbf{Set}$. The left adjoint of that functor is the functor $\mathbf{Set} \to G\textrm{-}\mathbf{Set}$ which gives each set a trivial $G$-action. Denote $X$ with a trivial $G$-action by $X_{triv}$. We have $\operatorname{Hom}_{G\textrm{-}\mathbf{Set}}(X,Y_{triv}) \cong \operatorname{Hom}_{\mathbf{Set}}(X/G,Y)$, so the functor which sends each $G$-set to the orbit space $X/G$ is left adjoint to the functor which gives each set a trivial $G$-action, so we can reconstruct the functor $X \mapsto X/G$ from the category $G\textrm{-}\mathbf{Set}$. Not sure if that's helpful.
Maybe it's even possible that $G$ and $H$ don't have to be isomorphic? I'm looking either for a counterexample or a proof.
 A: You can also invoke the following theorem : 

Two categories have equivalent categories of presheaves if and only if they have equivalent Cauchy completion.

Then remark that a group (seen as a category with one object) is Cauchy complete : its only idempotent is the neutral, which is obviously spit.
A: The empty $G$-set is the initial object.
The coproduct of $G$-sets is the disjoint union.
Call a $G$-set “indecomposable” if it is not the coproduct of two non-empty $G$-sets, so an indecomposable $G$-set is just a transitive one.
An epimorphism of $G$-sets is just a surjective map of $G$-sets.
Up to isomorphism, there is a unique transitive $G$-set with an epimorphism to every other transitive $G$-set, namely the regular $G$-set, whose automorphism group is $G$.
So $G$ can be recovered from the category of $G$-sets.
A: I think you already know this, but for the sake of completeness let me show how to prove the claim using your categorical approach.
For any category $\mathcal C$ equivalent to the category of left $G$-sets for some group $G$ we can reconstruct the forgetful functor up to noncanonical natural isomorphism:
The forgetful functor $F : G\text{-}\mathbf{Set} \to \mathbf{Set}$ is naturally isomorphic to $\mathrm{Hom}_{\mathbf{Set}}(\{*\}, F(-))$. It has a left adjoint given by the free $G$-set, that is the disjoint union $Fr(X) := \coprod_{x \in X} G$, where $G$ denotes the regular $G$-set. This shows, that $F \cong \mathrm{Hom}_{G\text{-}\mathbf{Set}}(G, -)$.
As Rickard explained the regular $G$-set is up to non-unique isomorphism characterized as the object, that admits an epimorphism to every connected nonempty $G$-set.
Now take such an object $X$. The functor $\mathrm{Hom}_{\mathcal C}(X,-) : \mathcal C \to \mathbf{Set}$ is after composition with the equivalence $\mathcal C \simeq G\text{-}\mathbf{Set}$ naturally isomorphic to the forgetful functor.
Yoneda gives a canonical isomorphism
$$ \mathrm{End}(F) \cong \mathrm{Hom}_{G\text{-}\mathbf{Set}}(G,G) \cong G $$
So altogether we have a non-canonical isomorphism $\mathrm{End}(\mathrm{Hom}_{\mathcal C}(X,-)) \cong G$.
