It is usually said that groups can (or should) be thought of as "symmetries of things". The reason is that the "things" which we study in mathematics usually form a category and for every object $X$ of a (locally small) category $\mathcal{C}$, the set of automorphisms (symmetries) of $X$, denoted by $\text{Aut}_{\mathcal{C}}(X)$, forms a group.
My question is: Which categories that occur naturally in mathematics admit all kinds of symmetries? More precisely, for which categories we can solve the equation (of course up to isomorphism) $$\text{Aut}_{\mathcal{C}}(X) = G$$ for every group $G$?
I will write what I could find myself about this, which also hopefully illustrates what kind of answers that would interest me: